Skip to content

โœจ feat(agent): support explicit registry credentials - #4839

Merged
mudler merged 1 commit into
kairos-io:masterfrom
Brilly-Bohyun:feat/registry-auth
Oct 16, 2026
Merged

mudler merged 1 commit into
kairos-io:masterfrom
Brilly-Bohyun:feat/registry-auth

Conversation

@Brilly-Bohyun

Copy link
Copy Markdown
Contributor

What this PR does / why we need it:

Docker and Podman credential files already support private registry authentication through the default keychain. This change adds an optional way to provide explicit credentials through install.registry-auth and upgrade.registry-auth, while preserving that existing behavior.

  • Add Auth to OCIImageExtractor and pass it to both image extraction and size lookup.
  • Apply registry settings before image sizing in regular and UKI install/upgrade flows.
  • Preserve authentication when combined with allow-insecure-registries.
  • Use the default keychain when explicit credentials are absent. Explicit credentials take precedence, and authentication failures do not fall back to the keychain.
  • Add configuration validation, schema definitions, examples, and credential redaction in config diagnostics.

Supported credential forms are username/password, base64 auth, identity-token, and registry-token.

This does not introduce secure credential storage or create Docker/Podman login files. Credentials supplied in cloud-config may be persisted through the existing installation flow.

Which issue(s) this PR fixes:
Fixes #2613

@ci-robbot ci-robbot left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this, and for reading root_schema_readers_test.go and adding the upgrade entry instead of working around it.

Built the head locally and ran sdk/schema, sdk/collector, sdk/utils/image and agent/pkg/config with CGO_ENABLED=0: all green. CI here is parked on authorize, so that is the only test evidence so far.

One thing I would fix before merge.

The redactor and the reader disagree on the key name

redactConfigValues normalises _ to - before matching, so it redacts registry_auth. readRegistryOptions looks up registry-auth only. A config written with the underscore spelling gets no credentials, no error, and a debug log showing [REDACTED] exactly where the credentials would be, which reads like they were picked up.

Probe against your head, agent/pkg/config:

Values: collector.ConfigValues{
    "upgrade": collector.ConfigValues{
        "registry_auth": collector.ConfigValues{"username": "bob", "password": "UNDERSCORESECRET"},
    },
}

RedactedConfigDump hides UNDERSCORESECRET; applyRegistryOptions(cfg, "upgrade") then returns Auth=<nil>.

That strings.ReplaceAll(key, "_", "-") is the only _/- folding in the repo (git grep finds it in scan.go and nowhere else), so the redactor is the outlier. Simplest fix is to drop the normalisation and match registry-auth exactly, which is the single spelling the schema declares.

The same gap exists from the other side: parseRegistryAuth rejects an unknown field inside the block, but nothing rejects a misspelt block name. registy-auth: or registry_auth: under install is accepted by the schema (it sets no additionalProperties) and then silently does nothing. Since you already fail the operation on a bad credential form, failing it on an unrecognised *auth* key under install/upgrade would be consistent.

Smaller notes, none blocking

  • applyRegistryOptions type-switches only on the value OCIImageExtractor, while RedactedConfigDump handles the pointer as well. agent/pkg/config/config.go:51 only ever stores the value, so the pointer branch is dead. Worth picking one.
  • The credentials stay on cfg.ImageExtractor for the rest of the process, so a sysext pulled later through agent/pkg/extensions/install.go sends the install credentials to whatever registry that extension names. Insecure already behaves this way so it is not a regression, but it deserves a line in examples/registry-auth.md.
  • RedactedConfigDump still prints users[].passwd in full. Pre-existing and out of scope, but it is the same dump and one more key in the same list.

The core plumbing looks right to me: GetImage already took an *registrytypes.AuthConfig and both extractor call sites (extract and GetOCIImageSize) are covered, so there is no path left pulling with nil auth.

type RegistryAuthSchema struct {
Username string `json:"username,omitempty" description:"Registry username. Must be provided with password."`
Password string `json:"password,omitempty" description:"Registry password."`
Auth string `json:"auth,omitempty" description:"Base64 encoded username:password."`

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what I would suggest is also to take these creds from files. In this way the user doesn't have to stamp the secrets in the cloud config, but they can make sure the file with credentials is present

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @mudler, added registry-auth.file for both install and upgrade.
It reads credentials from a YAML file, so cloud-config only needs the file path. Inline credentials remain supported, but combining them with file is rejected.
The file must be available before image sizing, which runs before the install and upgrade hooks.
Added documentation and tests covering regular and UKI install and upgrade flows.

@ci-robbot ci-robbot left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Confirmed fixed on 9b6d74a4: upgrade.registry_auth now fails with unsupported authentication key; use registry-auth instead of being redacted and ignored. agent/pkg/config, sdk/schema, sdk/collector and sdk/utils/image are green with CGO_ENABLED=0, and the file path reads credentials correctly.

One hole is left, and it is worse than the original because the secret reaches the log. validateRegistryAuthKeys matches strings.Contains(key, "auth"), so a typo inside "auth" dodges the guard, and redactConfigValues only redacts the exact key registry-auth, so the same typo dodges the redactor too. Probed against your head with the password SENTINEL under upgrade:

key error Auth SENTINEL in RedactedConfigDump
registry_auth rejected nil no
regisrty-auth rejected nil no
registry-auht none nil yes
registry-atuh none nil yes

That dump goes to Logger.Debugf("Loaded config: ...") in Scan, and install logs get copied onto the installed system, so a transposed typo prints the registry password in plaintext there.

Making the redactor a superset of the guard closes the class: redact any key whose lowercased form contains auth, passwd, password, token or secret rather than the one exact spelling. That also picks up the users[].passwd gap I mentioned before, which still prints in full.

Two smaller things from the same run, neither blocking:

  • registry-Auth and REGISTRY-AUTH are now rejected, but viper lowercases keys, so before d7f42cbb they worked. The error says "use registry-auth" to someone who did, modulo case. Folding the case in the guard before comparing would keep the strictness without the false positive.
  • Field names are case-insensitive inline (viper lowercases) but case-sensitive in the credential file (yaml.NewDecoder does not), so Username: in the file fails with registry-auth contains an unsupported field, which does not name the field. Worth naming the key in that error at least.

@ci-robbot

Copy link
Copy Markdown
Collaborator

Verified ae86a825 on a local build: the guard/redactor asymmetry is gone. registry-auht and registry-atuh now pass the guard, yield Auth: nil, and the password is [REDACTED] in the dump, and users[].passwd / p2p.network_token are redacted too, so the redactor is a proper superset of the guard now.

One over-redaction: isSensitiveConfigKey matches the substring auth, so users[].ssh_authorized_keys prints [REDACTED]. Public keys are not secrets and "did my key get through" is a common thing to check in a debug dump, so it is worth exempting authorized (or allowlisting that key) before this merges.

@ci-robbot

Copy link
Copy Markdown
Collaborator

Triage: this now conflicts with master (mergeable_state: dirty). @Brilly-Bohyun a rebase is needed before the checks can run again; the review feedback above is otherwise addressed.

@itxaka-agent

Copy link
Copy Markdown
Collaborator

Automated triage agent (kairos-triage-agent) running as @itxaka-agent.
This comment was generated by software. A human maintainer can override any
action taken here.

Reviewing this PR now: reading the registry-auth change end to end and checking the credential handling and redaction paths.

@itxaka-agent itxaka-agent left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated triage agent (kairos-triage-agent) running as @itxaka-agent.
This comment was generated by software. A human maintainer can override any
action taken here.

Changes requested: 2 blocking findings, both inline. The change touches install and upgrade, so it also needs a QEMU verification (authenticated install and upgrade, plus a boot with a misspelled key) once these are fixed.

cfg.Logger.Infof("Allowing insecure registry pulls via %s.allow-insecure-registries", subkey)
extractor.Insecure = true
}
if auth != nil {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The explicit credentials are stored on the shared extractor with no registry scope, so every later pull in the process sends them to whatever registry it contacts. install.extensions: [oci:quay.io/...] is pulled through the same extractor (hooks/extensions.go:154 and :248, via extensions/install.go:171), so the private registry's username and password go to quay.io or ghcr.io, and the extension pull can fail because that registry rejects them. Checked with go-containerregistry v0.22.1: static remote.WithAuth credentials were sent as Basic auth to the token endpoint of an unrelated registry, and the intended registry saw no request.

Scope the credentials to the registry host of the operation's source image (install.system.source or install.source for install, upgrade.system.source or upgrade.recovery-system.source for upgrade). For example, keep the host next to Auth on OCIImageExtractor and use the static auth in GetImage only when the parsed reference's Context().RegistryStr() matches it. Use authn.DefaultKeychain for every other registry. Add a test that pulls from a second httptest registry and asserts it never receives the first registry's Authorization header. Then drop the paragraph in examples/registry-auth.md that says extension pulls reuse the credentials.

Comment thread agent/pkg/config/scan.go Outdated
}

result.Collector = *genericConfig
if err := validateRegistryAuthKeys(result.Collector.Values); err != nil {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Calling validateRegistryAuthKeys inside scan() makes config.Scan return an error for every caller whenever a key under install or upgrade contains auth and is not registry-auth, for example upgrade.registry_auth left in /oem. agent.Run (agent/internal/agent/agent.go:27 and :56) then returns early on boot, so the provider bootstrap and k3s never start. Reset, notify and the sysext/state commands fail the same way, though they never use registry credentials.

Remove the call at scan.go:70-72. readRegistryOptions already rejects the typo when an install or upgrade spec is built, so those operations still fail closed, and RedactedConfigDump already redacts any key containing auth. Change the rejects misspelled blocks during scanning before diagnostics test to assert that scan succeeds and the logs do not contain the secret.

itxaka-agent
itxaka-agent previously approved these changes Oct 7, 2026

@itxaka-agent itxaka-agent left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated triage agent (kairos-triage-agent) running as @itxaka-agent.
This comment was generated by software. A human maintainer can override
any action taken here.

Re-checked after the latest push. Both earlier concerns are fixed: credentials are now only sent to the registry of the source image (other registries use the default keychain), and a misspelled registry-auth key no longer fails config scanning at boot. Tests cover both. Approving; a quick install and upgrade check against an authenticated registry would still be worth doing in QA.

@itxaka-agent

itxaka-agent commented Oct 7, 2026 •

Copy link
Copy Markdown
Collaborator

Automated triage agent (kairos-triage-agent) running as @itxaka-agent.
This comment was generated by software. A human maintainer can override
any action taken here.

QA is on hold: this PR has merge conflicts with master and its CI has not been authorized to run yet.

@itxaka-agent

Copy link
Copy Markdown
Collaborator

Automated triage agent (kairos-triage-agent) running as @itxaka-agent.
This comment was generated by software. A human maintainer can override
any action taken here.

@Itxaka the CI for this PR is waiting on approval (authorize job / fork checks). Could you authorize the run so QA can pick it up once the conflicts are fixed?

@itxaka-agent

Copy link
Copy Markdown
Collaborator

Automated triage agent (kairos-triage-agent) running as @itxaka-agent.
This comment was generated by software. A human maintainer can override
any action taken here.

@Brilly-Bohyun CI is failing on this PR at ad8bb06: unit-tests / test and lint / lint / golangci_lint, see https://github.com/kairos-io/kairos/actions/runs/37599611183. Could you take a look so it can go through QA?

@Brilly-Bohyun

Copy link
Copy Markdown
Contributor Author

@itxaka-agent Thanks for flagging this. Fixed in afa9b49. A merge error had placed OEMFiles in the upgrade schema instead of the install schema, causing the test failures, along with a formatting issue.

The affected test suites and lint now pass locally. I've also updated against master and squashed the commits. Could you take another look for QA once CI passes?

@itxaka-agent

Copy link
Copy Markdown
Collaborator

Automated triage agent (kairos-triage-agent) running as @itxaka-agent.
This comment was generated by software. A human maintainer can override
any action taken here.

@Itxaka the CI for this PR is waiting on approval (the new head afa9b496 has a run in action_required). Could you authorize the run so it can go through QA?

@itxaka-agent

Copy link
Copy Markdown
Collaborator

Automated triage agent (kairos-triage-agent) running as @itxaka-agent.
This comment was generated by software. A human maintainer can override
any action taken here.

@Itxaka the CI for this PR is waiting on approval (the new head f84848a3 has a run in action_required). Could you authorize the run so it can go through QA?

@itxaka-agent

Copy link
Copy Markdown
Collaborator

Automated triage agent (kairos-triage-agent) running as @itxaka-agent.
This comment was generated by software. A human maintainer can override
any action taken here.

QA: pass at 678b4b00 (install path), details and proof: #2613 (comment)

@itxaka-agent

Copy link
Copy Markdown
Collaborator

Automated triage agent (kairos-triage-agent) running as @itxaka-agent.
This comment was generated by software. A human maintainer can override
any action taken here.

QA: pass at 678b4b0, now including the upgrade path. Details and proof: #2613 (comment)

Comment thread agent/pkg/config/registry_auth.go Outdated
}

func readRegistryAuth(cfg *sdkConfig.Config, raw interface{}, subkey string) (*registrytypes.AuthConfig, error) {
values, _ := raw.(map[string]interface{})

@mudler mudler Oct 13, 2026 •

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure why we are reading here from the raw interface rather than using the typed fields. Any specific reason?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I used the raw map to distinguish omitted fields from explicitly empty strings, such as password: "", and to validate field names and types without including credential values in errors. These checks could also be implemented with typed fields and strict decoding. Would you prefer that approach?

@mudler mudler Oct 14, 2026 •

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

to omit empty strings you can use pointers in structs. yes, I'd prefer the typed approach, thank you

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Refactored the auth handling to use typed fields with *string pointers, as suggested, while preserving validation for empty and invalid values.

Signed-off-by: Brilly-Bohyun <102973953+Brilly-Bohyun@users.noreply.github.com>
@ci-robbot

Copy link
Copy Markdown
Collaborator

My 2026-09-23 over-redaction finding is fixed at b5dcb6c5: isSensitiveConfigKey returns false for ssh_authorized_keys before the substring loop, with case-variant tests, so public keys still print in the dump.

The blocker now is CI, not the code: the branch was rebased and squashed to one commit after the QA: pass at 678b4b00, and both runs at the new head (PR, CodeQL) are sitting in action_required, so nothing but DCO has run on what would merge. @Itxaka could you authorize them?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat: Expose auth args for image extractor in agent

4 participants