An Argo CD ApplicationSet turns one template into many Applications, one per cluster, directory or config file. Adding a cluster to the fleet stops meaning another hand-written manifest. The generators are easy to read and easy to get wrong: a typo in a goTemplate key produces zero Applications, two ApplicationSets that render the same name fight over ownership, and the wrong delete command takes your workloads with it.
We ran every generator (list, cluster, Git directory, Git file, matrix and merge) against two registered spoke clusters, then broke each one on purpose to capture the real error, and timed a RollingSync rollout from Git push to Healthy. Every command and output below came from Argo CD 3.6.1 on three kind clusters (Kubernetes 1.37) in November 2026.
Map the hub and spoke layout
Argo CD runs on one cluster only, the hub. The prod and staging clusters run nothing from Argo CD: they are registered as cluster secrets on the hub, and the application controller talks to their API servers on port 6443. The ApplicationSet controller sits next to it and does one job, which is rendering Application objects from generators and keeping them in step with Git.

Keep that split in mind when something fails. If no Application appears, the problem is in the ApplicationSet (template, generator, ownership). If the Application exists but won’t sync, the problem is between the hub and the spoke (credentials, DNS, the manifests in Git), and the ApplicationSet has already done its part.
Build the lab: hub, prod and staging
The lab needs three clusters and a Git server the hub can reach, and nothing else. We used kind on one Ubuntu 24.04 VM with 8 vCPUs and 32 GB of RAM, which is a comfortable floor for three single-node clusters plus Gitea. Production spokes are whatever you already run; the hub only needs network access to each spoke’s API server. Create the clusters:
for c in hub prod staging; do kind create cluster --name $c --wait 120s; done
That took 2 minutes 12 seconds for all three. Each cluster runs as a Docker container on the shared kind network, which matters later when Argo CD needs to resolve the spoke API servers:
docker ps --format "table {{.Names}}\t{{.Image}}\t{{.Status}}"
All three control planes should be up on the same node image:
NAMES IMAGE STATUS
staging-control-plane kindest/node:v1.37.0 Up 31 seconds
prod-control-plane kindest/node:v1.37.0 Up About a minute
hub-control-plane kindest/node:v1.37.0 Up About a minute
For Git we ran Gitea in a container attached to the same network, so pods on the hub reach it as http://gitea:3000. Any Git server works here; the generators only care that the repo server can clone the URL. Start it:
docker run -d --name gitea --network kind -p 127.0.0.1:3000:3000 \
-e GITEA__security__INSTALL_LOCK=true -e GITEA__server__ROOT_URL=http://gitea:3000/ -e GITEA__database__DB_TYPE=sqlite3 \
gitea/gitea:latest
With INSTALL_LOCK set there is no web installer, so create the first user from inside the container. Wait for the health endpoint first, because the first start generates the config and migrates the SQLite database. Pick your own password and keep it in a shell variable, because the repo API call and the Git remote both need it:
for i in $(seq 40); do curl -fs http://127.0.0.1:3000/api/healthz >/dev/null && break; sleep 3; done
export GITEA_PASS='Use-A-Strong-Pass-1'
docker exec -u git gitea gitea admin user create --username josphat --password "$GITEA_PASS" --email [email protected] --admin --must-change-password=false
Then create an empty public repository called fleet through the Gitea API. Public matters here, since it lets Argo CD clone without a repository credential:
curl -s -u josphat:"$GITEA_PASS" -H 'Content-Type: application/json' -d '{"name":"fleet","private":false,"default_branch":"main"}' http://127.0.0.1:3000/api/v1/user/repos | jq -r .clone_url
The clone URL it prints, http://gitea:3000/josphat/fleet.git, is the address every ApplicationSet below uses. Now build the repo content locally. The workload is a plain nginx Deployment with a Service, laid out as a Kustomize base:
mkdir -p ~/fleet/apps/webapp/base ~/fleet/apps/webapp/overlays/{prod,staging} ~/fleet/addons/{logging,metrics} ~/fleet/clusters/{prod,staging}
vim ~/fleet/apps/webapp/base/deployment.yaml
Leave the base at one replica, because the overlays decide the real count per environment:
apiVersion: apps/v1
kind: Deployment
metadata:
name: webapp
labels:
app: webapp
spec:
replicas: 1
selector:
matchLabels:
app: webapp
template:
metadata:
labels:
app: webapp
spec:
containers:
- name: nginx
image: nginx:1.27-alpine
ports:
- containerPort: 80
Add the Service in ~/fleet/apps/webapp/base/service.yaml:
apiVersion: v1
kind: Service
metadata:
name: webapp
labels:
app: webapp
spec:
selector:
app: webapp
ports:
- port: 80
targetPort: 80
Next, ~/fleet/apps/webapp/base/kustomization.yaml just lists both files:
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- deployment.yaml
- service.yaml
Each overlay patches the replica count. This is ~/fleet/apps/webapp/overlays/prod/kustomization.yaml; the staging copy in overlays/staging is identical except for value: 2:
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- ../../base
patches:
- target:
kind: Deployment
name: webapp
patch: |-
- op: replace
path: /spec/replicas
value: 3
The add-ons are deliberately tiny, one ConfigMap each, so the generator behaviour is easy to see. This is ~/fleet/addons/logging/configmap.yaml; the metrics one only changes the name to metrics-config:
apiVersion: v1
kind: ConfigMap
metadata:
name: logging-config
data:
enabled: "true"
The two clusters/*/config.json files are one line each and are shown in the Git file generator section; create them now so the first commit is complete. Then commit everything and push it to Gitea. The remote URL stores the password in ~/fleet/.git/config, which is acceptable for a throwaway lab and not for anything else:
cd ~/fleet
git init -q -b main; git config user.name josphat; git config user.email [email protected]
git add -A; git commit -qm "initial fleet repo"
git remote add origin "http://josphat:${GITEA_PASS}@127.0.0.1:3000/josphat/fleet.git"
git push -u origin main
The repo now has this layout. The apps/webapp tree is the Kustomize base with one overlay per environment (3 replicas in prod, 2 in staging), addons holds one directory per add-on, and clusters holds one JSON file per cluster:
tree ~/fleet
Nine files in total:
/home/josphat/fleet
├── addons
│ ├── logging
│ │ └── configmap.yaml
│ └── metrics
│ └── configmap.yaml
├── apps
│ └── webapp
│ ├── base
│ │ ├── deployment.yaml
│ │ ├── kustomization.yaml
│ │ └── service.yaml
│ └── overlays
│ ├── prod
│ │ └── kustomization.yaml
│ └── staging
│ └── kustomization.yaml
└── clusters
├── prod
│ └── config.json
└── staging
└── config.json
13 directories, 9 files
Install Argo CD on the hub. The ApplicationSet controller ships in the standard install manifest, so there is nothing extra to enable for the basic generators. Our Argo CD install guide covers ingress and TLS; for this lab the plain manifest is enough:
kubectl config use-context kind-hub
kubectl create namespace argocd
kubectl apply -n argocd --server-side --force-conflicts -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml
kubectl -n argocd wait --for=condition=Ready pods --all --timeout=600s
Every pod was Ready 56 seconds after the apply. Check that argocd-applicationset-controller is among them:
kubectl -n argocd get pods
Seven pods, all Running:
NAME READY STATUS RESTARTS AGE
argocd-application-controller-0 1/1 Running 0 53s
argocd-applicationset-controller-794bb95c9d-mgqq6 1/1 Running 0 54s
argocd-dex-server-66cf64bfcf-g5l5r 1/1 Running 0 54s
argocd-notifications-controller-5878c8d644-4rgqh 1/1 Running 0 54s
argocd-redis-5869b7547-6brzg 1/1 Running 0 54s
argocd-repo-server-7d55cc466b-lzhzm 1/1 Running 0 54s
argocd-server-85b849c688-6zbcb 1/1 Running 0 53s
The rest of the guide uses the argocd CLI for cluster registration and argocd appset commands. If it isn’t on your workstation yet, install the Argo CD CLI first, then port-forward the API server and log in as admin (the CLI login guide covers SSO and tokens for anything beyond a lab):
nohup kubectl port-forward svc/argocd-server -n argocd 8080:443 --address 127.0.0.1 >~/pf.log 2>&1 &
kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath='{.data.password}' | base64 -d > ~/argocd-admin-pass.txt; chmod 600 ~/argocd-admin-pass.txt
argocd login 127.0.0.1:8080 --username admin --password "$(cat ~/argocd-admin-pass.txt)" --insecure
Once you’re logged in, argocd version reports both the client and the server build, which is the quickest way to confirm the CLI is talking to the hub and not some other context.
Register the spoke clusters
Register each spoke with argocd cluster add and label it, because every generator that targets clusters reads Argo CD’s cluster secrets and their labels. argocd cluster add logs into the spoke with your kubeconfig, creates an argocd-manager ServiceAccount there and stores its token on the hub. Export each spoke’s kubeconfig with its internal address, because the hub reaches the spokes over the kind network and not through localhost:
for c in prod staging; do kind get kubeconfig --name $c --internal > ~/$c.kubeconfig; done
argocd cluster add kind-prod --name prod --kubeconfig ~/prod.kubeconfig --yes
Our first attempt failed. The CLI also uses that kubeconfig from the VM itself, and the VM’s resolver has never heard of prod-control-plane:
{"level":"fatal","msg":"failed to create service account \"argocd-manager\" in namespace \"kube-system\": Post \"https://prod-control-plane:6443/api/v1/namespaces/kube-system/serviceaccounts\": dial tcp: lookup prod-control-plane on 127.0.0.53:53: server misbehaving","time":"2026-11-13T09:05:22Z"}
The fix is to make the name resolvable on the machine running the CLI, while keeping the URL Argo CD stores the same one the hub can resolve. On a kind lab that means adding each control plane’s container IP to /etc/hosts. On real clusters the same error means your kubeconfig points at a name only the cluster network knows, and you should use the API server’s routable DNS name instead:
for c in prod staging; do ip=$(docker inspect -f '{{(index .NetworkSettings.Networks "kind").IPAddress}}' $c-control-plane); echo "$ip $c-control-plane" | sudo tee -a /etc/hosts; done
argocd cluster add kind-prod --name prod --kubeconfig ~/prod.kubeconfig --yes
argocd cluster add kind-staging --name staging --kubeconfig ~/staging.kubeconfig --yes
This time both spokes register, and each one reports the ServiceAccount, ClusterRole and token it created:
{"level":"info","msg":"ServiceAccount \"argocd-manager\" created in namespace \"kube-system\"","time":"2026-11-13T09:05:27Z"}
{"level":"info","msg":"ClusterRole \"argocd-manager-role\" created","time":"2026-11-13T09:05:27Z"}
{"level":"info","msg":"ClusterRoleBinding \"argocd-manager-role-binding\" created","time":"2026-11-13T09:05:27Z"}
{"level":"info","msg":"Created bearer token secret \"argocd-manager-long-lived-token\" for ServiceAccount \"argocd-manager\"","time":"2026-11-13T09:05:27Z"}
Cluster 'https://prod-control-plane:6443' added
Now label them. The cluster generator selects on these labels and exposes them to templates, so treat them as the fleet’s inventory: env picks the overlay and tier marks how careful a rollout has to be.
argocd cluster set prod --label env=prod --label tier=critical
argocd cluster set staging --label env=staging --label tier=standard
kubectl -n argocd get secret -l argocd.argoproj.io/secret-type=cluster --show-labels
Labels land on the cluster secrets themselves, which is what the generator reads:
NAME TYPE DATA AGE LABELS
cluster-prod-control-plane-3648912337 Opaque 3 6s argocd.argoproj.io/secret-type=cluster,env=prod,tier=critical
cluster-staging-control-plane-1443425697 Opaque 3 5s argocd.argoproj.io/secret-type=cluster,env=staging,tier=standard
Don’t worry if argocd cluster list shows prod as Unknown with “Cluster has no applications and is not being monitored.” A cluster may sit in that state until an Application targets it. In our lab staging already read Successful at that point while prod read Unknown, and every sync to prod afterwards worked, so this state alone is not a registration failure.
Choose a generator
Pick the generator by where the list of targets already lives, because that source becomes the thing you edit to change the fleet:
- A short fixed list in your head: the list generator
- Argo CD’s cluster registry: the cluster generator
- The repo layout: a Git directory or file generator
- Two of those at once: matrix or merge
These are the times we measured from kubectl apply to every generated Application reporting Synced and Healthy, so you know what normal looks like before something goes wrong:
| Generator | Source of targets | Apps in our lab | Apply to Healthy |
|---|---|---|---|
| list | Elements written in the ApplicationSet | 2 | 10 s |
| clusters | Cluster secrets matching a label selector | 2 | 2 s |
| git directories | Directories matching a path glob | 2, then 3 | 3 s (new dir: 357 s) |
| git files | JSON or YAML files matching a glob | 2 | 3 s |
| matrix | Every combination of two generators | 6 | 12 s |
| merge | A base generator with per-key overrides | 2 | not timed (we were checking the tier labels, not health) |
One number stands out: 357 seconds. A Git directory generator without a webhook only notices a new directory on its next poll, and that delay is covered in the Git directories section below. Everything else was ready within seconds, because the generator output was already known when the ApplicationSet was applied.
Deploy with the list generator
The list generator is the right choice for a handful of fixed targets, and it’s the easiest one to debug because every parameter is visible in the manifest. Each element becomes one Application, and its keys become template variables. We keep all the manifests in ~/lab/manifests on the workstation, so create that directory and the first manifest:
mkdir -p ~/lab/manifests
vim ~/lab/manifests/webapp-list.yaml
Every ApplicationSet in this guide sets goTemplate: true with missingkey=error. Go templates give you default, get and other Sprig functions, and missingkey=error turns a typo into a visible error instead of an Application named webapp-<no value>:
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
name: webapp-list
namespace: argocd
spec:
goTemplate: true
goTemplateOptions: ["missingkey=error"]
generators:
- list:
elements:
- cluster: prod
url: https://prod-control-plane:6443
path: apps/webapp/overlays/prod
- cluster: staging
url: https://staging-control-plane:6443
path: apps/webapp/overlays/staging
template:
metadata:
name: 'webapp-{{.cluster}}'
spec:
project: default
source:
repoURL: http://gitea:3000/josphat/fleet.git
targetRevision: main
path: '{{.path}}'
destination:
server: '{{.url}}'
namespace: webapp
syncPolicy:
automated:
prune: true
selfHeal: true
syncOptions:
- CreateNamespace=true
Apply it on the hub, then ask Argo CD what it generated:
kubectl apply -f ~/lab/manifests/webapp-list.yaml
argocd appset get webapp-list
kubectl -n argocd get applications
Read the three conditions first. ParametersGenerated True means the generator produced its parameter sets, and ResourcesUpToDate True means the Applications match the template. The {{.url}} and {{.path}} shown in the Server and Path fields are normal: appset get prints the template, not the rendered values.

Both spokes got their overlay within 10 seconds of the apply: webapp ran 3/3 replicas on prod and 2/2 on staging. Note one more thing on the generated Application before moving on, because it decides what happens when you delete the ApplicationSet:
kubectl -n argocd get application webapp-prod -o jsonpath='{.metadata.finalizers}'
By default, the Application carries the resources finalizer:
["resources-finalizer.argocd.argoproj.io"]
With that finalizer present, kubectl -n argocd delete appset webapp-list removed both Applications and the webapp Deployment with them; on prod only the empty namespace was left behind. That’s the right default for a test, and a nasty surprise in production, so the deletion section further down shows how to keep workloads alive.
Target clusters by label with the cluster generator
The cluster generator removes the URL bookkeeping. It walks the cluster secrets on the hub, keeps the ones matching the selector, and hands each match to the template as .name, .server and .metadata.labels. Register a new cluster with env=prod and it gets the workload with no change in Git:
vim ~/lab/manifests/webapp-clusters.yaml
The overlay path is built from the cluster’s env label, so the label and the directory name have to agree:
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
name: webapp-clusters
namespace: argocd
spec:
goTemplate: true
goTemplateOptions: ["missingkey=error"]
generators:
- clusters:
selector:
matchExpressions:
- key: env
operator: In
values: [prod, staging]
template:
metadata:
name: 'webapp-{{.name}}'
spec:
project: default
source:
repoURL: http://gitea:3000/josphat/fleet.git
targetRevision: main
path: 'apps/webapp/overlays/{{.metadata.labels.env}}'
destination:
server: '{{.server}}'
namespace: webapp
syncPolicy:
automated:
prune: true
selfHeal: true
syncOptions:
- CreateNamespace=true
Apply it and check what the generator produced:
kubectl apply -f ~/lab/manifests/webapp-clusters.yaml
argocd appset get webapp-clusters
kubectl -n argocd get applications
The template fields still show {{.server}} and the label path, while all three conditions report success:
Name: argocd/webapp-clusters
Project: default
Server: {{.server}}
Namespace: webapp
Health Status: Healthy
Source:
- Repo: http://gitea:3000/josphat/fleet.git
Target: main
Path: apps/webapp/overlays/{{.metadata.labels.env}}
SyncPolicy: Automated (Prune)
CONDITION STATUS MESSAGE LAST TRANSITION
ErrorOccurred False All applications have been generated successfully 2026-11-13 09:06:41 +0000 UTC
ParametersGenerated True All parameters have been generated successfully 2026-11-13 09:06:41 +0000 UTC
ResourcesUpToDate True All applications have been generated successfully 2026-11-13 09:06:41 +0000 UTC
Both labelled spokes got an Application, Synced and Healthy 2 seconds after the apply:
NAME SYNC STATUS HEALTH STATUS
webapp-prod Synced Healthy
webapp-staging Synced Healthy
Keep the selector. We rendered a copy with clusters: {} and no selector through argocd appset generate, which previews the output without creating anything:
argocd appset generate ~/lab/manifests/webapp-clusters-noselector.yaml -o json | jq -r '.[].metadata.name'
Notice the hub itself showing up as a third target:
webapp-prod
webapp-staging
webapp-in-cluster
An empty cluster generator includes in-cluster, the hub Argo CD runs on. Applied for real, that would have deployed the prod overlay onto the management cluster. Run argocd appset generate before every new ApplicationSet; it costs nothing and catches exactly this kind of surprise.
Generate apps from Git directories
The Git directory generator creates one Application per directory matching a glob, which suits platform add-ons: drop a new directory under addons/ and it deploys. In goTemplate mode the path comes through as an object, so you write {{.path.path}} for the full path and {{.path.basename}} for the last segment:
vim ~/lab/manifests/addons-git-dir.yaml
To keep the example short, this one deploys to the hub (https://kubernetes.default.svc). The matrix section sends the same add-ons to the spokes:
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
name: addons-git-dir
namespace: argocd
spec:
goTemplate: true
goTemplateOptions: ["missingkey=error"]
generators:
- git:
repoURL: http://gitea:3000/josphat/fleet.git
revision: main
directories:
- path: addons/*
template:
metadata:
name: 'addons-{{.path.basename}}'
spec:
project: default
source:
repoURL: http://gitea:3000/josphat/fleet.git
targetRevision: main
path: '{{.path.path}}'
destination:
server: https://kubernetes.default.svc
namespace: 'addons-{{.path.basename}}'
syncPolicy:
automated:
prune: true
selfHeal: true
syncOptions:
- CreateNamespace=true
Applying it created addons-logging and addons-metrics in 3 seconds. Then we pushed a third add-on, addons/tracing/configmap.yaml, a copy of the logging ConfigMap named tracing-config, and waited. The matrix section later relies on it too:
cd ~/fleet; git add -A; git commit -qm "add tracing addon"; git push -q origin main
addons-tracing appeared 357 seconds after the push. The ApplicationSet controller polls Git every 3 minutes by default, and the Git generator also reads through the repo server’s revision cache, so without a webhook a new directory can sit unseen for more than one poll.
If six minutes is too slow, note that the Argo CD API server’s webhook does not help here. Per the Git generator docs, the ApplicationSet controller runs its own webhook server (the argocd-applicationset-controller Service, port 7000), which you expose with a separate Ingress and point GitHub or GitLab at its /api/webhook. We left the lab on polling, since Gitea is not one of the documented senders; requeueAfterSeconds on the generator is the documented knob for a shorter poll. To skip a directory instead, add a second path entry with exclude: true:
directories:
- path: addons/*
- path: addons/logging
exclude: true
Excluding a directory is a delete, not a pause. Applying the excluded version removed addons-logging and, because of the resources finalizer, its ConfigMap went with it:
kubectl -n argocd get applications
Only two add-ons remain:
NAME SYNC STATUS HEALTH STATUS
addons-metrics Synced Healthy
addons-tracing Synced Healthy
Use exclude when you really want an add-on gone from every cluster. If you only want to stop syncing it for a while, set preserveResourcesOnDeletion on the ApplicationSet first.
Read per-cluster values with the Git file generator
The Git file generator reads JSON or YAML files and turns every top-level key into a template variable. It’s the cleanest way to keep per-cluster settings in Git, reviewed like any other change, instead of in cluster secret labels. Each cluster gets one small file:
cat ~/fleet/clusters/prod/config.json ~/fleet/clusters/staging/config.json
Each file holds the cluster name, its API URL and a few settings:
{"cluster":"prod","url":"https://prod-control-plane:6443","replicas":"3","tier":"critical"}
{"cluster":"staging","url":"https://staging-control-plane:6443","replicas":"2","tier":"standard"}
Point the generator at the glob and use the keys directly. Create the manifest:
vim ~/lab/manifests/webapp-git-files.yaml
Here {{.cluster}} and {{.url}} come straight from the JSON:
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
name: webapp-git-files
namespace: argocd
spec:
goTemplate: true
goTemplateOptions: ["missingkey=error"]
generators:
- git:
repoURL: http://gitea:3000/josphat/fleet.git
revision: main
files:
- path: clusters/*/config.json
template:
metadata:
name: 'webapp-{{.cluster}}'
spec:
project: default
source:
repoURL: http://gitea:3000/josphat/fleet.git
targetRevision: main
path: 'apps/webapp/overlays/{{.cluster}}'
destination:
server: '{{.url}}'
namespace: webapp
syncPolicy:
automated:
prune: true
selfHeal: true
syncOptions:
- CreateNamespace=true
Apply it, then check the ApplicationSet, the Applications and the Deployment on each spoke:
kubectl apply -f ~/lab/manifests/webapp-git-files.yaml
argocd appset get webapp-git-files
kubectl -n argocd get applications
kubectl --context kind-prod -n webapp get deploy
kubectl --context kind-staging -n webapp get deploy
Here Server reads {{.url}}, the key from the JSON files, and the conditions are clean:
Name: argocd/webapp-git-files
Project: default
Server: {{.url}}
Namespace: webapp
Health Status: Healthy
Source:
- Repo: http://gitea:3000/josphat/fleet.git
Target: main
Path: apps/webapp/overlays/{{.cluster}}
SyncPolicy: Automated (Prune)
CONDITION STATUS MESSAGE LAST TRANSITION
ErrorOccurred False All applications have been generated successfully 2026-11-13 09:13:52 +0000 UTC
ParametersGenerated True All parameters have been generated successfully 2026-11-13 09:13:52 +0000 UTC
ResourcesUpToDate True All applications have been generated successfully 2026-11-13 09:13:52 +0000 UTC
Both Applications were Synced and Healthy 3 seconds after the apply, and each overlay landed with its own replica count, 3/3 on prod and 2/2 on staging:
NAME SYNC STATUS HEALTH STATUS
webapp-prod Synced Healthy
webapp-staging Synced Healthy
NAME READY UP-TO-DATE AVAILABLE AGE
webapp 3/3 3 3 2s
NAME READY UP-TO-DATE AVAILABLE AGE
webapp 2/2 2 2 2s
The trade-off against the cluster generator is that the URL now lives in two places, the cluster secret and the JSON file, so a re-registered cluster with a new address needs a commit too.
Combine generators with matrix
Matrix produces every combination of two child generators. Git directories times clusters is the classic use: every add-on on every cluster, so three add-ons on two clusters give six Applications. Parameters from both children are available, which is why the name below mixes .path.basename from Git with .name from the cluster:
vim ~/lab/manifests/addons-matrix.yaml
Paste the matrix definition:
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
name: addons-matrix
namespace: argocd
spec:
goTemplate: true
goTemplateOptions: ["missingkey=error"]
generators:
- matrix:
generators:
- git:
repoURL: http://gitea:3000/josphat/fleet.git
revision: main
directories:
- path: addons/*
- clusters:
selector:
matchExpressions:
- key: env
operator: In
values: [prod, staging]
template:
metadata:
name: '{{.path.basename}}-{{.name}}'
spec:
project: default
source:
repoURL: http://gitea:3000/josphat/fleet.git
targetRevision: main
path: '{{.path.path}}'
destination:
server: '{{.server}}'
namespace: addons
syncPolicy:
automated:
prune: true
selfHeal: true
syncOptions:
- CreateNamespace=true
Apply it and list the result:
kubectl apply -f ~/lab/manifests/addons-matrix.yaml
kubectl -n argocd get applications
Six Applications, all Synced and Healthy 12 seconds after the apply:
NAME SYNC STATUS HEALTH STATUS
logging-prod Synced Healthy
logging-staging Synced Healthy
metrics-prod Synced Healthy
metrics-staging Synced Healthy
tracing-prod Synced Healthy
tracing-staging Synced Healthy
In the web UI, the Applications page shows the same six tiles, each pointing at its own add-on path and destination cluster:

Watch the multiplication. Matrix output grows as the product of its children, so ten add-ons on twenty clusters is two hundred Applications, and every one of them is reconciled by the same application controller. Check the count with argocd appset generate before you apply a matrix to a real fleet.
Override one cluster with the merge generator
Merge takes a base generator and lets later generators override parameters for entries that share a merge key. The usual job is “every cluster gets the default, prod gets something different”. It is also the generator where we hit the most surprises, so here is what worked and what didn’t. The version that works keeps the override as a plain top-level key:
vim ~/lab/manifests/webapp-merge-toplevel.yaml
The list element matches the prod cluster on server and adds tier: critical. Clusters without an override fall back to standard through get and default, which also keeps missingkey=error happy:
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
name: webapp-merge-toplevel
namespace: argocd
spec:
goTemplate: true
goTemplateOptions: ["missingkey=error"]
generators:
- merge:
mergeKeys:
- server
generators:
- clusters:
selector:
matchExpressions:
- key: env
operator: In
values: [prod, staging]
- list:
elements:
- server: https://prod-control-plane:6443
tier: critical
template:
metadata:
name: 'webapp-{{.name}}'
labels:
tier: '{{ get . "tier" | default "standard" }}'
spec:
project: default
source:
repoURL: http://gitea:3000/josphat/fleet.git
targetRevision: main
path: 'apps/webapp/overlays/{{.metadata.labels.env}}'
destination:
server: '{{.server}}'
namespace: webapp
syncPolicy:
automated:
prune: true
selfHeal: true
syncOptions:
- CreateNamespace=true
Apply it and check which tier label each Application received:
kubectl apply -f ~/lab/manifests/webapp-merge-toplevel.yaml
kubectl -n argocd get applications -o custom-columns=NAME:.metadata.name,TIER:.metadata.labels.tier,REV:.spec.source.targetRevision,PATH:.spec.source.path,SYNC:.status.sync.status,HEALTH:.status.health.status
Prod picked up the override and staging kept the default:
NAME TIER REV PATH SYNC HEALTH
webapp-prod critical main apps/webapp/overlays/prod Synced Healthy
webapp-staging standard main apps/webapp/overlays/staging Synced Healthy
Two variants we tried first did not work. Putting the cluster defaults under the cluster generator’s values: block and overriding them with flat values.tier: critical keys in the list (the form the official merge example uses) applied cleanly but changed nothing: both Applications came out tier=standard. Nesting a values: map inside the list element was worse, and the merge section of the troubleshooting list below shows the controller panic it caused.
Roll out in stages with RollingSync
Every generator so far syncs all clusters at once, so a bad commit reaches prod and staging together. RollingSync, part of the Progressive Syncs feature, changes that: it groups the generated Applications into ordered steps by label and only syncs the next step once the previous one is Healthy. The upstream Progressive Syncs documentation marks it as a Beta feature, and it is off by default. Turn it on in argocd-cmd-params-cm and restart the controller:
kubectl -n argocd patch cm argocd-cmd-params-cm --type merge -p '{"data":{"applicationsetcontroller.enable.progressive.syncs":"true"}}'
kubectl -n argocd rollout restart deploy argocd-applicationset-controller
kubectl -n argocd rollout status deploy argocd-applicationset-controller --timeout=120s
The ConfigMap key only takes effect through an environment variable the controller reads at startup, which is why the restart matters. Confirm the running pod picked it up:
kubectl -n argocd exec deploy/argocd-applicationset-controller -- env | grep PROGRESSIVE
Expect true:
ARGOCD_APPLICATIONSET_CONTROLLER_ENABLE_PROGRESSIVE_SYNCS=true
Now define the steps. Each generated Application gets an envLabel label from the list element, and the two steps select staging first, then prod. Create the manifest:
vim ~/lab/manifests/webapp-rolling.yaml
We left automated in the template on purpose to see what RollingSync does with it:
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
name: webapp-rolling
namespace: argocd
spec:
goTemplate: true
goTemplateOptions: ["missingkey=error"]
generators:
- list:
elements:
- cluster: staging
envLabel: staging
url: https://staging-control-plane:6443
path: apps/webapp/overlays/staging
- cluster: prod
envLabel: prod
url: https://prod-control-plane:6443
path: apps/webapp/overlays/prod
strategy:
type: RollingSync
rollingSync:
steps:
- matchExpressions:
- key: envLabel
operator: In
values:
- staging
- matchExpressions:
- key: envLabel
operator: In
values:
- prod
template:
metadata:
name: 'webapp-{{.cluster}}'
labels:
envLabel: '{{.envLabel}}'
spec:
project: default
source:
repoURL: http://gitea:3000/josphat/fleet.git
targetRevision: main
path: '{{.path}}'
destination:
server: '{{.url}}'
namespace: webapp
syncPolicy:
automated:
prune: true
selfHeal: true
syncOptions:
- CreateNamespace=true
RollingSync overrode it. Both generated Applications came out with automated: map[enabled:false prune:true selfHeal:true], so the controller owns every sync from then on. The docs say the controller logs a warning when the template asks for autosync; we grepped the controller log and found none, so don’t count on a warning to tell you. The Application page shows it plainly instead:

- 1Auto sync is not enabled
- 2Progressive Sync step
- 3Owning ApplicationSet
The Sync Status panel reads “Auto sync is not enabled” (1), the Progressive Sync step panel (2) shows which step this Application belongs to and whether it finished, and the Owning ApplicationSet badge in the resource tree (3) links back to webapp-rolling. That badge is the quickest way to find which ApplicationSet to edit when someone asks why a manual change to an Application keeps reverting.
A health gate only means something if health takes a moment to arrive, so before the timed run we added a readiness probe to the container in apps/webapp/base/deployment.yaml, right under its ports list, and pushed it:
readinessProbe:
httpGet:
path: /
port: 80
initialDelaySeconds: 15
To time a real rollout, we then pushed a commit at 09:28:04 that bumped the image from nginx:1.27-alpine to nginx:1.28-alpine, and polled the Applications every 2 seconds, logging each state change. These are the logged changes:
| Time (UTC) | staging (step 1) | prod (step 2) | Image on prod |
|---|---|---|---|
| 09:28:05 | Healthy, Synced | Healthy, Synced | nginx:1.27-alpine |
| 09:29:27 | Healthy, Synced | Waiting, OutOfSync | nginx:1.27-alpine |
| 09:29:58 | Progressing, sync running | Waiting, OutOfSync | nginx:1.27-alpine |
| 09:30:44 | Healthy | Progressing, Synced | nginx:1.28-alpine |
| 09:31:50 | Healthy | Healthy | nginx:1.28-alpine |
Read the table as the controller’s gate at work. Prod had gone OutOfSync by the 09:29:27 sample and then sat in Waiting while staging synced; the rollout moved to step 2 at 09:30:42, the moment staging reported Healthy. The whole rollout took 226 seconds from push to prod Healthy, and the 15 second readiness delay we had added to the Deployment is part of that, because the gate waits for health, not for the sync to finish. The ApplicationSet records each step itself:

While a rollout runs, the RolloutProgressing condition reads “ApplicationSet is performing rollout of step 1”, then step 2. When it ends you get “ApplicationSet Rollout has completed”, the same text the ApplicationSet page shows under its health:

Two more settings are worth knowing. maxUpdate on a step caps how many Applications in that step sync at once and defaults to 100%, which is fine for one cluster per step and too aggressive for a step of twenty. deletionOrder: Reverse under strategy deletes in the opposite order of the steps; we tested it on a copy of this ApplicationSet and the prod Application was removed at 09:33:36, staging at 09:33:37, with “completed reverse deletion for ApplicationSet webapp-rolling-reverse” in the log. If you want the same staged pattern inside a single cluster at the pod level, that’s what Argo Rollouts canary deployments are for; RollingSync stages clusters, not replicas.
Delete an ApplicationSet without deleting workloads
Set preserveResourcesOnDeletion: true before you need it. As the Application deletion docs explain, generated Applications get the resources finalizer unless that flag is set, and the finalizer is what tells Argo CD to delete everything the Application deployed. The flag sits under the ApplicationSet’s own syncPolicy, not the template’s:
spec:
syncPolicy:
preserveResourcesOnDeletion: true
goTemplate: true
goTemplateOptions: ["missingkey=error"]
We tested three ways out, starting each time from the same two-cluster list ApplicationSet:
- Flag added to a running ApplicationSet: finalizers removed from both Applications, workloads survived
- Flag set from the first apply: no finalizer ever added, workloads survived
- No flag, deleted with
--cascade=orphan: Applications kept, finalizer kept
The first two behave the way you’d hope, so adding the flag to an existing ApplicationSet is safe and immediate. The third is the trap. --cascade=orphan looks like the safe delete, and at first it is: the Applications stay and the workloads keep running. But the orphaned Applications still carry the finalizer, so the next person who deletes one of them deletes its workload too:

Deleting webapp-staging removed the staging Deployment, while prod kept running only because nobody had touched its Application yet. Re-applying the same ApplicationSet adopted the orphans again with the finalizer still in place, and it also recreated webapp-staging and redeployed it. So if you’ve orphaned Applications by accident, re-adopt them and add preserveResourcesOnDeletion: true before the next delete. We ran those two halves separately (the first test shows the flag dropping finalizers on a live ApplicationSet), not as one sequence, so check the finalizers with kubectl -n argocd get applications -o custom-columns=NAME:.metadata.name,FINALIZERS:.metadata.finalizers before you delete anything.
Fix common ApplicationSet errors
Every error below came from a manifest we broke on purpose. In most cases the ApplicationSet showed the reason in its conditions and created nothing new (the panic case is the exception, with no conditions at all), so kubectl -n argocd get appset <name> -o jsonpath='{.status.conditions}' | jq . is the first command to run when nothing appears.
map has no entry for key
The template referenced {{.nonexistent}}, a key the list element doesn’t define. With missingkey=error the controller refuses to render, marks the ApplicationSet Degraded and reports reason RenderTemplateParamsError:
failed to execute go template webapp-{{.nonexistent}}: template: base:1:9: executing "base" at <.nonexistent>: map has no entry for key "nonexistent"
Fix the key name, or wrap optional keys as {{ get . "key" | default "x" }} like the merge example does. The corrected manifest generated webapp-prod and the conditions flipped back to “All applications have been generated successfully”.
function “path” not defined
This one comes from copying an old fasttemplate example into a goTemplate ApplicationSet. {{path.basename}} without the leading dot is fasttemplate syntax, and Go templates read path as a function call:
failed to parse template addons-{{path.basename}}: template: base:1: function "path" not defined
Add the dot and use the object form: {{.path.basename}} for the directory name and {{.path.path}} for the full path. Plain {{path}} also has to become {{.path.path}}, since .path on its own is now an object.
contains applications with duplicate name
A template with a fixed name (name: webapp) and two list elements renders two Applications with the same name. The controller logs “generated 2 applications”, then rejects the whole set with reason ApplicationValidationError and creates neither:
ApplicationSet dup-names contains applications with duplicate name: webapp
Put a per-target parameter in the name, such as webapp-{{.cluster}}. With a matrix, include a parameter from each child generator, or two combinations will collide.
is already owned by another ApplicationSet controller
Two different ApplicationSets that render the same Application name don’t merge or overwrite each other. The first one keeps it and the second one goes into error with UpdateApplicationError:

When we deleted dup-a, dup-b adopted webapp-prod on its next reconcile. That handover can surprise you during a migration from one ApplicationSet to another: the new one silently takes over, with its own sync policy, as soon as the old one is gone. Give each ApplicationSet a distinct name prefix and the error never happens.
Recovered from panic: reflect.Value.SetMapIndex
This came from the merge generator with a nested values: map inside a list element. The ApplicationSet showed an empty health status and generated nothing, while the controller log filled with the same panic, 1,634 times by the time we counted:
kubectl -n argocd logs deploy/argocd-applicationset-controller | grep webapp-merge-nested | grep -o '"msg":"Recovered from panic: [^\\]*' | sort | uniq -c
A count that high means the controller is retrying in a tight loop:
1634 "msg":"Recovered from panic: reflect.Value.SetMapIndex: value of type interface {} is not assignable to type string
Delete that ApplicationSet straight away, because the loop burns controller CPU for every other ApplicationSet too, and move the override to a top-level key as shown in the merge section.
lookup on 127.0.0.53:53: server misbehaving
This is the argocd cluster add failure from the registration step. The machine running the CLI can’t resolve the API server name in the kubeconfig. Make the name resolvable there, or regenerate the kubeconfig with an address both the CLI and the hub can reach, because the URL you register is the one Argo CD uses for every sync afterwards.
Harden it before the fleet grows
Put these six guardrails in place before a second team writes an ApplicationSet. Everything above ran as admin in the default project, which is fine for a lab and wrong for a shared hub:
goTemplate: truewithmissingkey=erroron every ApplicationSetpreserveResourcesOnDeletion: trueon anything serving traffic- A selector on every cluster generator, so
in-clusteris never a target by accident - A webhook to the ApplicationSet controller, not argocd-server, for Git generators
- One AppProject per team, with allowed destinations limited to their clusters
- RollingSync with
maxUpdateset on any step that spans many clusters
Once several generators are live, the ApplicationSets page is the first sanity check after any generator change. Each tile shows the health and the number of Applications it owns:

The AppProject point matters most, because an ApplicationSet can render any destination its template allows, and the project is the only thing stopping a typo from targeting the wrong cluster. The Argo CD CLI reference covers argocd proj for setting that up, and if you’re still deciding whether Argo CD is the right hub at all, our Flux vs Argo CD multi-cluster comparison runs the same kind of fleet on both. Our verdict for most teams: start with the cluster generator plus labels, add matrix for add-ons, and switch to RollingSync the day prod and staging stop being allowed to change together.