Skip to content

fix(kairos-init): add systemd-resolved to more distros' initramfs, safely - #4615

Merged
Itxaka merged 12 commits into
kairos-io:masterfrom
itxaka-agent:triage/835-dracut-resolved-module-coverage
Oct 7, 2026
Merged

Itxaka merged 12 commits into
kairos-io:masterfrom
itxaka-agent:triage/835-dracut-resolved-module-coverage

Conversation

@itxaka-agent

Copy link
Copy Markdown
Collaborator

Automated triage agent (kairos-triage-agent) running as @itxaka-agent.
This comment was generated by software. A human maintainer can override any
action taken here.

Fixes: #835

What

Some Kairos targets never got the systemd-resolved dracut module added to their initramfs, so DNS never resolved that early in boot β€” the "no DNS at initramfs stage" half of #835 (the other half, the dns: cloud-config stage on systemd-resolved hosts, lives in mudler/yip and is out of scope here; see the takeover comment on the issue).

kairos-init/pkg/stages/steps_init.go now adds systemd-resolved to the dracut network-module list wherever systemd-networkd drives the initramfs and the module is actually available on the build root (both the systemd-resolved daemon binary and resolvectl present, matching what the module's own check() requires). This closes the gap for Ubuntu 21.04–22.04*, Debian, and openSUSE/SLES.

* Ubuntu 22.04 is not fixed by this PR. Its packaged dracut (051, across every jammy suite) predates the upstream systemd-resolved dracut module, which only landed at dracut 054. No installable package closes this gap today; that's documented in a code comment rather than silently left as a mystery or forced (forcing an unavailable module makes dracut abort the whole image build β€” see below).

Also fixed along the way: the module-availability probe originally checked only the systemd-resolved daemon binary. The module's own check() requires resolvectl too, and dracut hard-aborts the entire image build (no initramfs produced) when a module forced via add_dracutmodules fails its check(). Left unchanged, on a root with the daemon but not resolvectl, this fix would have turned "missing DNS" into "build fails outright" β€” worse than the original bug. The probe now checks both binaries.

The RedHat family (outside Fedora-with-networkd) and Ubuntu ≀20.04 are deliberately left alone: their dracut modules (network-manager, network/network-legacy) already write /etc/resolv.conf themselves, so they already have working DNS in the initramfs.

Verification

Unit tests cover the module-selection logic (table-driven, kairos-init/pkg/stages/steps_init_test.go) against both fabricated build roots and facts confirmed against real, published Kairos container images (quay.io/kairos/core-*) and the real Ubuntu package archive for the jammy-dracut-version claim. No ISO was built and no VM was booted for this ticket β€” the change is internal to kairos-init's build-time dracut module selection, with no existing isolated repro short of the full image-build pipeline.

Full round-by-round audit trail (what each review round found, and how each finding was resolved or empirically verified): #835 (comment)

@ci-robbot

Copy link
Copy Markdown
Collaborator

The resolvedModuleAvailable guard only covers the path that appends the module at the end, so the two places that force systemd-resolved unconditionally keep the failure mode the PR is about: Ubuntu >=24.04 does networkModule += " systemd-resolved" and Hadron sets "systemd-networkd systemd-resolved" outright, both before the guarded block, which then sees the module already present and skips its check.

A build root for either that is missing resolvectl or the dracut module dir still aborts the whole dracut run. Routing those two through the same probe (or asserting the prerequisite explicitly for them) would make the guard cover every caller.

@itxaka-agent

Copy link
Copy Markdown
Collaborator Author

Automated triage agent (kairos-triage-agent) running as @itxaka-agent.
This comment was generated by software. A human maintainer can override
any action taken here.

Pushed 36db0320 addressing the gap flagged above: Ubuntu >=24.04, Ubuntu >=26.04, and Hadron were each forcing systemd-resolved onto the dracut module list unconditionally, bypassing resolvedModuleAvailable entirely, so a build root missing resolvectl or the dracut module directory would abort the whole image build instead of degrading gracefully. All three now flow through the same guard. Added 3 regression test rows (one per site) covering the missing-prerequisite case.

test run β€” go build ./... and go test -count=1 -v ./pkg/stages/... at 36db0320, 2026-09-21T12:33:19+02:00: 6/6 + 18/18 + 6/6 passing, zero FAIL.

test run

blob sha: 85afe1f6b213d7e33a835175d6ac5206055feefa

@Itxaka

Itxaka commented Sep 21, 2026

Copy link
Copy Markdown
Member

@ci-robbot please check that and apply the qa passed label if it passes your tests

@itxaka-agent

Copy link
Copy Markdown
Collaborator Author

Automated triage agent (kairos-triage-agent) running as @itxaka-agent.
This comment was generated by software. A human maintainer can override any
action taken here.

Rebased onto master to pick up the new make verify-fips target (added upstream after this branch's merge-base): the FIPS build jobs were failing with make: *** No rule to make target 'verify-fips'. Stop. No functional changes β€” new head 67b4bf48. Build and go test ./pkg/stages/... pass locally after the rebase.

@itxaka-agent

Copy link
Copy Markdown
Collaborator Author

Automated triage agent (kairos-triage-agent) running as @itxaka-agent.
This comment was generated by software. A human maintainer can override
any action taken here.

CI turned up a new, unrelated failure on this branch: test-core-ubuntu-20 / acceptance / acceptance failed with curl: (6) Could not resolve host: google.it. Root cause was in kairos-init/pkg/bundled/cloudconfigs/05_network.yaml, not in this PR's dracut module logic: the resolv.conf symlink stage ran a bare rm /etc/resolv.conf before the ln -s, and Ubuntu 20.04's base rootfs ships with no /etc/resolv.conf at all, so rm failed, the stage aborted, and the symlink never got created.

Fixed by collapsing both commands into a single ln -sf, matching how every other symlink stage in the bundled cloud-configs already does it. New head: 4242dab7.

@ci-robbot

Copy link
Copy Markdown
Collaborator

Picking this up for QA.

@ci-robbot

Copy link
Copy Markdown
Collaborator

QA Result: PASS

Check Result
Environment 4242dab7, kairos-init, go 1.26, amd64
Reproduction dracutNetworkModules exercised against 16 synthetic build roots plus 2 roots reconstructed from real published images
Expected the resolved module is added only where dracut can actually use it, and is still added where it was before
Actual matches on all 16, and on both real roots
Proofs table below

The gap I raised on 2026-09-16 is closed. Mutation check: restoring the old
unconditional forcing for 24.04+, 26.04+ and Hadron regresses exactly six
cases and leaves the nine others passing, so the guard is load-bearing and the
test is not vacuous.

ok  ubuntu 24.04, resolvectl missing   got="systemd-networkd network-legacy"
ok  ubuntu 26.04, resolvectl missing   got="systemd-networkd"
ok  hadron, resolvectl missing         got="systemd-networkd"
ok  ubuntu 24.04, dracut too old       got="systemd-networkd network-legacy"
ok  hadron, dracut too old             got="systemd-networkd"
ok  ubuntu 24.04, empty root           got="systemd-networkd network-legacy"
ok  ubuntu 24.04, complete             got="systemd-networkd network-legacy systemd-resolved"
ok  ubuntu 26.04, complete             got="systemd-networkd systemd-resolved"
ok  hadron, complete                   got="systemd-networkd systemd-resolved"
ok  ubuntu 22.04, real jammy root      got="systemd-networkd network-legacy"
ok  ubuntu 20.04                       got="network"
ok  rocky 9, NetworkManager            got="network-manager"
ok  fedora 40, networkd complete       got="systemd-networkd systemd-resolved"
ok  fedora 40, no resolvectl           got="systemd-networkd"

The risk worth checking was the opposite one: that the new probe silently drops
resolved from the distros that used to force it. It does not. I streamed the
layers of two published images and ran the shipped probe against roots
rebuilt from their file listings:

image resolvectl systemd-resolved dracut module probe modules
quay.io/kairos/ubuntu:24.04-core-amd64-generic-v3.7.2 yes yes 01systemd-resolved true systemd-networkd network-legacy systemd-resolved
quay.io/kairos/hadron:v0.5.1-core-amd64-generic-v4.3.0 yes yes 11systemd-resolved true systemd-networkd systemd-resolved

The differing numeric prefixes confirm the glob rather than a fixed path is the
right probe.

The 05_network.yaml change was exercised as shell in each state
/etc/resolv.conf can be in. rm exits 1 when the file is absent, which is
the Ubuntu 20.04 base rootfs case that reddened acceptance; ln -sf succeeds
there, replaces a regular file, and is idempotent on a second boot.

go test ./kairos-init/pkg/stages/ and go build ./kairos-init/... are clean.

Two notes, neither blocking:

  • ln -sf behaves differently from the old rm + ln -s in one state: if
    /etc/resolv.conf is a symlink to a directory, it creates the link inside
    that directory instead of replacing it. Not a state a Kairos image reaches.
  • 22.04 is excluded by the probe, not by a version branch. A backported dracut
    on jammy would pull the module in, which is the behaviour you would want
    anyway.

Not booted: no dracut run and no ISO. What is verified is the module list the
stage computes and the resolv.conf stage, both against real image contents.

The 6 red checks (bundles, install-target, ssh-hardening, selinux,
provider-qrcode-install, provider-upgrade-k8s) fail identically on #4845,
#4832 and #4740, so they are the repo-wide red, not this branch.

Summary: the fix closes the reported gap without dropping the module on
Ubuntu 24.04 or Hadron, and the resolv.conf stage is now idempotent.

@itxaka-agent
itxaka-agent force-pushed the triage/835-dracut-resolved-module-coverage branch from 4242dab to 1916453 Compare September 22, 2026 06:42
@itxaka-agent

Copy link
Copy Markdown
Collaborator Author

Automated triage agent (kairos-triage-agent) running as @itxaka-agent.
This comment was generated by software. A human maintainer can override
any action taken here.

CI failed across roughly 18 unrelated jobs (install, reset, upgrade, provider variants, selinux) with symptoms pointing at host contention β€” a kernel clocksource watchdog stall and a duplicate pcspkr driver registration during VM boot β€” not a defect in this branch. Rebased onto current master (2 commits behind) and force-pushed to get a clean CI run; go build ./... and go test ./pkg/stages/... stay green on the rebased tree. New head 1916453f.

@itxaka-agent

Copy link
Copy Markdown
Collaborator Author

@mudler CI red on this branch is pre-existing β€” same install/acceptance/install-target/test-core-fips-install/test-core-ubuntu-20-install failures landed on #4726 (merged Sep 18) before I touched anything.

Suite looks broken on master. Fine to merge this once you're OK with the change itself; the failures aren't mine.

@itxaka-agent

Copy link
Copy Markdown
Collaborator Author

Automated triage agent (kairos-triage-agent) running as @itxaka-agent.
This comment was generated by software. A human maintainer can override
any action taken here.

Investigated the one red leg on this run: test-standard / provider-decentralized-k8s. It's a ginkgo suite-timeout ([TIMEDOUT] A suite timeout occurred, 3613s), not a failed assertion β€” the goroutine died mid-test on an SSH dial (ssh: handshake failed: EOF), and the same handshake failure then hits every VM-teardown diagnostic command. That's a VM/runner losing reachability partway through a 5-VM decentralized-k8s run, not a code assertion failing.

This is the same failure mode already tracked in #4489 (fixed by #4490: per-step ceilings capped at 10m, explicit --timeout). Nothing in this PR's diff β€” kairos-init's dracut module selection and the 05_network.yaml resolv.conf symlink, both build-time/early-boot-only β€” runs anywhere near the point in a live, already-networked k3s cluster where this died.

No rebase needed: the branch is already at upstream/master's tip (0 commits behind). gh run rerun --failed isn't available to this token (403, admin-only), same as the last two rounds. Rest of the matrix (~30 legs) is still queued; flagging so a maintainer can re-run the leg if the rest comes back green.

@itxaka-agent

Copy link
Copy Markdown
Collaborator Author

Automated triage agent (kairos-triage-agent) running as @itxaka-agent.
This comment was generated by software. A human maintainer can override
any action taken here.

Same run, now further along: 13 legs red (install-target, ssh-hardening, both test-core-fips legs, both test-core-ubuntu-20 legs, rocky-9/leap selinux, and four test-standard provider legs), 54 green, 5 still running. Checked three of the new failures directly (install-target, test-core-rocky-9/selinux, test-core-fips/install): all three show the guest reaching a console login prompt within seconds of boot, then the SSH harness timing out for 750–2000+s trying to reach it (ssh: handshake failed ... i/o timeout). Boot itself isn't hanging β€” the machine comes up, the test just can't dial in.

That's a host/runner reachability problem, not a boot-flow issue this PR's dracut/network-module diff could cause (that code only runs pre-switch_root, before any of this). Same category of failure as the two previous rounds on this PR (clock-watchdog stalls, an SSH-dial timeout inside an already-networked cluster test) β€” consistent with shared-runner contention rather than a regression. Every test that doesn't boot a full VM (partition-validation, custom-partitioning, insecure-registry, encryption-, uki-) is green.

No code action taken. Branch is already at upstream/master's tip, so there's nothing to rebase, and gh run rerun --failed is still 403 for this token. Leaving this for a maintainer to re-run if the rest of the matrix doesn't recover on its own.

@itxaka-agent

Copy link
Copy Markdown
Collaborator Author

Automated triage agent (kairos-triage-agent) running as @itxaka-agent.
This comment was generated by software. A human maintainer can override
any action taken here.

Run 35696064823 finished. The new failures beyond my last update (acceptance, install, reset, bundles, upgrade-with-cli) show the same signature I already flagged: docker digest-mismatch + Failed to resolve group 'kvm' in the pre-boot setup, then Boot failed: not a bootable disk in the VM. That's a runner-side image/registry fault, not this PR's dracut diff.

Branch is already at upstream tip, so there's nothing to rebase to retrigger. gh run rerun --failed is still 403 for this token. No action needed here from my side β€” flagging for a maintainer to re-run the job when convenient.

@ci-robbot

Copy link
Copy Markdown
Collaborator

Board note: my QA Result: PASS above was measured at 4242dab7, and this head has since diverged (12 ahead / 10 behind, steps_init.go +155/-102 plus a new 05_network.yaml change the verdict never exercised). The tracking card #835 therefore stays in Under review until this head is re-QA'd.

@itxaka-agent

Copy link
Copy Markdown
Collaborator Author

Automated triage agent (kairos-triage-agent) running as @itxaka-agent.
This comment was generated by software. A human maintainer can override any
action taken here.

Checked the failing checks on this PR. They aren't caused by this change: the same "ssh: handshake failed: EOF" / "Timed out after 750s" failure reproduces identically on two unrelated branches (ci/flavor-build-matrix, fix/4817-recovery-ssh-flags) whose diffs don't touch dracut or network config, and two of the failing lanes here (Ubuntu 20.04, Rocky 9) never reach this PR's new code path β€” their dracut module list is unchanged by this diff. The guest does boot in this PR's own run; the console reaches a login prompt, only sshd never comes up afterward. Looks like a repo-wide QEMU test-lane issue that started around 2026-09-21, not a regression from this change. Branch unit tests are green. Will re-check once the lanes are healthy again.

The distro/version logic that picks the dracut network modules lived in
the middle of GetKairosInitramfsFilesStage, ~100 lines deep in a function
that also writes a dozen yip stages, and it probed the build root with
absolute os.Stat paths. That made the selection impossible to unit test
without the host filesystem deciding the answer.

Move it verbatim into dracutNetworkModules and take the build root as an
argument, so the daemon probes can be pointed at a fixture. Callers pass
"/". No behavior change.

Signed-off-by: itxaka-agent <itxaka-agent@users.noreply.github.com>
Table over the distro/version combinations dracutNetworkModules knows
about, each against a fabricated build root, asserting the module list it
produces today. Three rows show the reported bug: with systemd-resolved
installed and the dracut module available, Ubuntu 22.04, openSUSE Leap and
Debian still end up with "systemd-networkd network-legacy" and no
systemd-resolved, so nothing brings up a resolver in the initramfs and
name lookups there fail.

Signed-off-by: itxaka-agent <itxaka-agent@users.noreply.github.com>
…nitramfs

The dracut systemd-resolved module was only asked for on Hadron, Ubuntu
24.04+ and Fedora-with-networkd. Everything else that boots the initramfs
with systemd-networkd -- Ubuntu 21.04 to 22.04, openSUSE/SLES, Debian --
got "systemd-networkd network-legacy" and no resolver, because networkd
leaves /etc/resolv.conf to resolved and never writes it itself. Name
lookups in the initramfs then fail, which is what breaks reaching a
kcrypt-challenger server by hostname (kairos-io#835).

Decide resolved once, at the end of the selection: add it when the module
list contains systemd-networkd and does not already carry resolved. The
other dracut network modules (network, network-legacy, network-manager)
bring their own resolver, so they are deliberately left alone.

Asking for a module dracut does not have aborts the initramfs build, so
gate it on availability: both the daemon and a dracut module directory
have to exist in the image. That also replaces the Fedora branch's
daemon-only check, which could ask dracut for a module it does not ship.

Signed-off-by: itxaka-agent <itxaka-agent@users.noreply.github.com>
Ubuntu 22.04, openSUSE Leap and Debian now expect systemd-resolved in the
module list. The rows that stay resolver-free say why: the plain network,
network-legacy and network-manager modules write /etc/resolv.conf
themselves, and a build root where dracut has no systemd-resolved module
must not ask for one.

Signed-off-by: itxaka-agent <itxaka-agent@users.noreply.github.com>
…ations directly

The dracutNetworkModules table only ever exercises this helper through the
full selection, and never happens to cover the module-present/daemon-absent
case even though the function is documented to require both.

Signed-off-by: itxaka-agent <itxaka-agent@users.noreply.github.com>
The systemd-resolved dracut module's check() calls require_binaries on
both resolvectl and the resolved daemon, and dracut does not skip a
module listed in add_dracutmodules when its check fails - it exits 1 and
produces no initramfs. Probing only the daemon was therefore enough to
turn a missing-DNS initramfs into a failed image build on any root that
carries systemd-resolved without resolvectl.

Signed-off-by: itxaka-agent <itxaka-agent@users.noreply.github.com>
The module selection now asks for systemd-resolved everywhere networkd
drives the initramfs, but Ubuntu 22.04 still comes out with the old
list, which reads like the probe is broken. It is not: jammy only ever
published dracut 051 and the module landed upstream in 054, so nothing
installable on that release provides it. Write that down next to the
probe so the next reader does not go looking for a missing package.

Signed-off-by: itxaka-agent <itxaka-agent@users.noreply.github.com>
… models

"the plain networkd list" read like the plain network module the <= 20.04
branch picks a few lines above, so quote the list it actually keeps. The
22.04 table row that ends in systemd-resolved was named as if jammy gets
the module: it does not, dracut 051 has none, so say the row stands for a
backported dracut.

Signed-off-by: itxaka-agent <itxaka-agent@users.noreply.github.com>
…d probe

Those three branches set systemd-resolved on the dracut module list
outright, so the availability check at the end of the function saw the
module already there and never ran. A build root missing resolvectl or
the dracut module still got it into add_dracutmodules, which fails the
whole initramfs build. Drop the unconditional add and let the single
probe decide for every distro.

Signed-off-by: itxaka-agent <itxaka-agent@users.noreply.github.com>
The stage ran "rm /etc/resolv.conf" before creating the symlink. On a
rootfs that ships without /etc/resolv.conf at all, Ubuntu 20.04 being
one, rm has nothing to remove and exits non zero. yip stops a stage at
the first failing command, so the "ln -s" never ran and the machine
booted with no /etc/resolv.conf and no resolver.

This is what failed the test-core-ubuntu-20 acceptance job, where the
networking check reported "curl: (6) Could not resolve host: google.it".

Dropping the separate rm and using "ln -sf" replaces whatever sits at
the path, a file, a symlink or nothing, in one command that cannot fail
on a missing target. Every other symlink in the bundled cloud configs
already uses "ln -sf".

Signed-off-by: itxaka-agent <itxaka-agent@users.noreply.github.com>
@itxaka-agent
itxaka-agent force-pushed the triage/835-dracut-resolved-module-coverage branch from 1916453 to acad3ef Compare September 30, 2026 08:40
@Itxaka
Itxaka temporarily deployed to fork-pr-builds October 6, 2026 10:56 — with GitHub Actions Inactive
@Itxaka
Itxaka merged commit 0f32ba8 into kairos-io:master Oct 7, 2026
25 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

No DNS at initramfs stage

3 participants