Skip to content

fix(kairos-init): install the iSCSI initiator on the Red Hat family - #4740

Merged
mudler merged 1 commit into
kairos-io:masterfrom
ci-forks:fix/rhel-iscsi-initiator
Sep 24, 2026
Merged

mudler merged 1 commit into
kairos-io:masterfrom
ci-forks:fix/rhel-iscsi-initiator

Conversation

@ci-robbot

Copy link
Copy Markdown
Collaborator

Fixes #4739

kairos-init/pkg/bundled/cloudconfigs/09_systemd_services.yaml enables iscsid
in the initramfs stage of every systemd image. The Debian, SUSE and Alpine
families all install open-iscsi for it. The Red Hat family installed nothing:
the RPM is called iscsi-initiator-utils, and no package map named it.

The gap is in the shipped images, measured

Streamed both layers of the published v3.7.2 amd64 rootfs of four flavors and
listed every path matching iscsi:

image iscsid / iscsiadm iscsid.service /etc/iscsi
kairos/rockylinux:9-core-amd64-generic-v3.7.2 no no no
kairos/fedora:40-core-amd64-generic-v3.7.2 no no no
kairos/debian:13-core-amd64-generic-v3.7.2 yes, usr/sbin/iscsid yes yes
kairos/opensuse:leap-15.6-core-amd64-generic-v3.7.2 yes, sbin/iscsid yes yes

On Rocky and Fedora the only matches are kernel modules and dracut's own
95iscsi module, which require_binaries iscsiadm and is therefore skipped.

Why it matters

The failing systemctl enable is the least of it, since run-stage is
non-strict and swallows it. Booting from an iSCSI root on a Red Hat-family image
is impossible. And iSCSI-backed storage does not work at all: Longhorn and
Portworx both need iscsiadm and a running iscsid in the host namespace,
while 00_rootfs.yaml persists /etc/iscsi, /var/lib/longhorn and /etc/pwx
for exactly that case.

The intent is already in the tree elsewhere. RedHatFamily installs
device-mapper-multipath and nfs-utils for storage parity, #3670 asked for
open-iscsi and it was added to Hadron in kairos-io/hadron#59, and
kairos-init/pkg/stages/steps_init.go:1130 says the enable exists so that "the
post-boot iscsid service stays enabled for anyone actually using iSCSI storage".

Scope of the package entry

ArchAMD64 and ArchARM64, not ArchCommon. Every RPM mirror is unreachable
from where I work, so I cannot confirm the package exists for Fedora riscv64,
and guessing an entry there is how a build breaks for one arch only. Those two
arches cover every published Red Hat-family Kairos image. This mirrors how
open-vm-tools is already split for the SUSE family.

CI's build-iso / amd64-core-rocky-9 job builds from rockylinux/rockylinux:9
on every pull request, so the amd64 half is checked here. test-core-rocky-9
then boots it.

Test

kairos-init/pkg/validation/cloudconfigs_systemd_services_test.go reads the
unit list out of the cloud-config rather than retyping it, so adding a unit
there reaches the spec, and maps each unit to its package per family rather than
assuming the unit and the package share a name (they do for logrotate, they do
not for the iSCSI initiator).

Mutation-checked both halves separately:

  • without the fix at all: fedora/amd64 installs no iscsi-initiator-utils, but 09_systemd_services.yaml enables iscsid on every boot
  • with only the ArchAMD64 entry: fedora/arm64 installs no iscsi-initiator-utils, ...
  • with both: ok github.com/kairos-io/kairos/v4/kairos-init/pkg/validation

go test ./kairos-init/... passes, gofmt and golangci-lint are clean on the
two touched packages.

Not verified by a boot. This changes the package list, not the boot path, and
the Rocky leg of the ISO build plus test-core-rocky-9 cover it in CI.


AI was used to write this change. No human has read it yet.

@ci-robbot

Copy link
Copy Markdown
Collaborator Author

The red qemu matrix here is master, not this PR: the same 18 jobs fail on master run 35345388402 and this branch only touches kairos-init. Tracked in #4743.

@ci-robbot ci-robbot moved this from Under review πŸ” to QA in πŸ§™Issue tracking board Sep 18, 2026
@ci-robbot

Copy link
Copy Markdown
Collaborator Author

Picking this up for QA.

@ci-robbot

Copy link
Copy Markdown
Collaborator Author

QA Result: PASS

Check Result
Environment PR head d69bbaa7. Two Rocky 9 OS images built from images/Dockerfile with BASE_IMAGE=rockylinux/rockylinux:9, differing only in kairos-init: kairos-init:master-scratch (baseline) vs kairos-init:pr-4740. ISOs from quay.io/kairos/auroraboot:v0.27.1 build-iso, booted under qemu-kvm (-cpu host).
Reproduction Baseline vs PR image contents; live boot of both ISOs; automated install to disk from the PR ISO and boot of the installed node; real iSCSI LUN attach from the installed node against a tgtd target on the host; go test ./kairos-init/... plus a mutation check; values.GetPackages() diff over 8 distro/arch pairs.
Expected Red Hat-family images install iscsi-initiator-utils, so the systemctl enable iscsid in 09_systemd_services.yaml succeeds, dracut stops skipping its 95iscsi module, and iSCSI-backed storage works. No other family or arch changes.
Actual All of that holds on the PR image and none of it on the baseline. Details below.
Proofs https://gist.github.com/ci-robbot/e2d3527fbed28e608068df9c1211241d (image A/B, both boots, the iSCSI attach transcript, tests + mutation check, package-map diff, serial logs)

What the baseline does (master-built Rocky 9 ISO, live boot):

immucore[593]: * failed to run systemctl enable iscsid: exit status 1
$ sudo iscsiadm --version   -> sudo: iscsiadm: command not found
$ systemctl is-enabled iscsid -> Failed to get unit file state ... No such file or directory

What the PR does, installed to disk and booted from disk:

KAIROS_VERSION="v0.0.0-qa4740-pr"     systemctl is-system-running -> running, 0 failed units
systemctl is-enabled iscsid -> enabled     is-active -> active     iscsiadm version 6.2.1.11
immucore: no "failed to run systemctl enable iscsid" line

The three impacts in #4739, each checked separately:

  1. The failing enable is gone. The stage now creates /etc/systemd/system/multi-user.target.wants/iscsid.service at boot.
  2. dracut no longer skips 95iscsi: lsinitrd -m /boot/initrd lists the iscsi module and the initrd carries usr/sbin/iscsiadm + usr/sbin/iscsid. The baseline initrd has none of them. The image still boots clean both live and installed, so adding the module did not reintroduce the Ubuntu-20-style dependency cycle on Rocky.
  3. iSCSI-backed storage works. Against a userspace tgtd target on the host (128M LUN, iqn.2026-09.qa4740:lun0), the installed node ran discovery, logged in, got sda 128M disk iscsi VIRTUAL-DISK, mounted it, read back data written during the earlier live boot, appended, remounted and read it back again. On the baseline this is not reachable at all, there is no iscsiadm.

Scope and regressions:

  • values.GetPackages() diffed across ubuntu/amd64, debian/arm64, opensuse-leap/amd64, alpine/amd64, fedora amd64+arm64+riscv64 and rocky/amd64: exactly three added lines, all iscsi-initiator-utils on Red Hat-family amd64 and arm64. Fedora riscv64 is untouched, as the PR describes.
  • go test ./kairos-init/... passes at PR head. The new spec is not vacuous: reverting only the packagemaps.go hunk and keeping the test fails with fedora/amd64 installs no iscsi-initiator-utils, but 09_systemd_services.yaml enables iscsid on every boot.

One unrelated observation, not caused by this PR and not present on the baseline path being tested here: the installed node still logs failed to run systemctl enable systemd-confext: exit status 1 from the same immucore stage. Same class as this issue, different unit. Reporting it, not fixing it.

Summary: The Red Hat family now ships the iSCSI initiator. The boot-time enable succeeds, the dracut module is included, and an iSCSI volume attaches and carries a filesystem on a real installed Rocky 9 node, none of which works on a master-built image. No change to any other family or to riscv64.

@ci-robbot ci-robbot closed this Sep 20, 2026
@ci-robbot ci-robbot reopened this Sep 20, 2026
@github-project-automation github-project-automation Bot moved this from QA to Done βœ… in πŸ§™Issue tracking board Sep 20, 2026
@github-project-automation github-project-automation Bot moved this from Done βœ… to Under review πŸ” in πŸ§™Issue tracking board Sep 20, 2026
@ci-robbot

Copy link
Copy Markdown
Collaborator Author

Retriggered with a close/reopen. The 09-18 matrix went red on every flavor including ubuntu-20 and leap, which this Red Hat package map cannot touch, and that run's log and artifact blobs now 403 on age, so a fresh run is the only way to see what failed.

@ci-robbot ci-robbot moved this from Under review πŸ” to QA in πŸ§™Issue tracking board Sep 20, 2026
@ci-robbot

Copy link
Copy Markdown
Collaborator Author

Picking this up for QA.

Re-QA: this PR was moved back into QA at 06:04 UTC today, after my 09-18 PASS and after the close/reopen that retriggered CI. Head is unchanged at d69bbaa7, so this cycle covers the fresh CI run plus a re-verification against current master.

@ci-robbot

Copy link
Copy Markdown
Collaborator Author

QA Result: PASS

Check Result
Environment PR head d69bbaa7 cherry-picked onto current master d7f3b5e1 (clean, the PR was one commit behind). Two Rocky 9 OS images built from images/Dockerfile with BASE_IMAGE=rockylinux/rockylinux:9, differing only in the kairos-init image: CI's own ghcr.io/kairos-io/kairos/kairos-init:pr-4740 (built 2026-09-18T14:29 from this head) vs :master. ISO from quay.io/kairos/auroraboot:v0.27.1, installed to disk and booted under qemu-kvm with -cpu host.
Reproduction Image A/B on iSCSI contents, go test ./kairos-init/... plus a mutation check on the merged tree, values.GetPackages() diff over 8 distros x 3 arches, live-CD autoinstall, boot of the installed node, and a real iSCSI LUN attach from that node against a tgtd target on the host.
Expected Red Hat family images install iscsi-initiator-utils, the systemctl enable iscsid in 09_systemd_services.yaml succeeds, dracut stops skipping 95iscsi, and iSCSI backed storage works. Nothing else changes.
Actual All of it holds on the PR image and none of it on the baseline. No regression against current master.
Proofs https://gist.github.com/ci-robbot/4d1da108067785112c896d3c391ad5c6 (image A/B, both build logs, live and installed serial logs, the iSCSI attach transcript, tests and mutation check, package-map diff, CI state)

Re-QA of the same head as my 2026-09-18 PASS, so this run re-checked it against master as it is today and against the artifact CI actually publishes.

Image A/B, same Dockerfile, same base, only kairos-init differs:

:pr-4740 :master
iscsiadm / iscsid /usr/sbin/iscsiadm, /usr/sbin/iscsid missing
iscsid.service present missing
/etc/iscsi present missing
rpm iscsi-initiator-utils-6.2.1.11-1.git4b3e853.el9_8.2.x86_64 not installed
dracut 95iscsi in initrd included skipped

The published kairos-init binaries agree: strings | grep -c iscsi-initiator-utils is 1 for :pr-4740 and 0 for :master.

Installed node, booted from disk after an autoinstall from the PR ISO:

KAIROS_VERSION=v0.0.0-qa4740b-pr
KAIROS_INIT_VERSION=v4.3.0-118-gd69bbaa7
systemctl is-system-running : running        systemctl --failed : 0 units
is-enabled iscsid           : enabled        is-active iscsid   : active
iscsiadm                    : iscsiadm version 6.2.1.11
enable symlink              : /etc/systemd/system/multi-user.target.wants/iscsid.service -> /usr/lib/systemd/system/iscsid.service

KAIROS_INIT_VERSION naming this PR's commit is what ties the booted system to the change.

Real iSCSI attach from that node to a 128 MiB file-backed tgtd LUN on the host: discovery, login, sda 128M disk iscsi VIRTUAL-DISK, mkfs.ext4, mount, write, umount, remount, read back, logout. The string written inside the guest is then greppable in the host's LUN backing file, so the data really crossed the wire. None of this is reachable on the baseline, there is no iscsiadm.

Scope, values.GetPackages() diffed across ubuntu, debian, fedora, rocky, almalinux, rhel, opensuse-leap and alpine on amd64, arm64 and riscv64: exactly 8 added lines, all iscsi-initiator-utils on Red Hat family amd64 and arm64. riscv64 and every other family untouched.

Tests: go test ./kairos-init/... passes on the merged tree. The new spec is not vacuous there either: reverting only the packagemaps.go hunk fails with Bundled systemd services cloudconfig [It] installs the package behind every unit it enables at cloudconfigs_systemd_services_test.go:106.

On the CI retrigger. The close/reopen created run 35488430428, and it has never executed: its state is waiting, gated at authorize. So the red qemu jobs still shown on this PR are the 2026-09-18 run 35355275138, which is the master-wide regression in #4743 (yip v1.26.2). Master is still d7f3b5e1 with github.com/mudler/yip v1.26.2, and #4744, the bump that fixes it, is stuck at the same authorize gate. A maintainer click on either unblocks the signal; nothing here is this PR's doing.

Unchanged from last cycle and unrelated: the installed node still logs failed to run systemctl enable systemd-confext from the same immucore stage. Same class, different unit. Reporting it, not fixing it.

Summary: The Red Hat family ships the iSCSI initiator, the boot-time enable succeeds with zero failed units, the dracut module is included, and an iSCSI volume attaches and carries a filesystem on a real installed Rocky 9 node. None of that works on a master-built image, and nothing else in the package map moves. The fresh CI run is blocked on the workflow approval gate, not on this change.

`09_systemd_services.yaml` enables `iscsid` in the initramfs stage of every
systemd image. The Debian, SUSE and Alpine families all install `open-iscsi`,
and the Red Hat family installed nothing: the RPM is called
`iscsi-initiator-utils`, and no package map named it.

Streaming both layers of the published v3.7.2 amd64 rootfs shows the gap is
real, not a naming difference in the image:

    rockylinux:9-core   no iscsid, no iscsiadm, no iscsid.service, no /etc/iscsi
    fedora:40-core      the same
    debian:13-core      usr/sbin/iscsid, usr/lib/systemd/system/iscsid.service
    opensuse:leap-15.6  sbin/iscsid, usr/lib/systemd/system/iscsid.service

On Rocky and Fedora the only matches are kernel modules and dracut's own
95iscsi module, which `require_binaries iscsiadm` and is therefore skipped.

Three consequences, in increasing order of how much they matter. The enable
fails on every boot, which `run-stage` swallows. Booting from an iSCSI root is
impossible. And iSCSI backed storage does not work at all: Longhorn and
Portworx both need `iscsiadm` and a running `iscsid` in the host namespace,
while `00_rootfs.yaml` persists `/etc/iscsi`, `/var/lib/longhorn` and
`/etc/pwx` for exactly that case.

Listed per architecture rather than under `ArchCommon` because the package
cannot be confirmed for Fedora riscv64 from here, the same reason
`open-vm-tools` is split that way for the SUSE family. These two arches cover
every published Red Hat family Kairos image, and CI's
`build-iso / amd64-core-rocky-9` job checks the amd64 half on every pull
request.

The guard reads the unit list out of the cloud-config instead of retyping it,
so adding a unit there reaches the spec, and it maps each unit to its package
per family rather than assuming the two share a name.

Fixes kairos-io#4739

Signed-off-by: Ettore Di Giacinto <mudler@kairos.io>
@mudler
mudler merged commit 9c88456 into kairos-io:master Sep 24, 2026
105 of 143 checks passed

This branch was previously deployed

1 inactive deployment
fork-pr-builds β€” cca62095 Deployed Sep 21, 2026 by ci-robbot via authorize #672
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

kairos-init: Red Hat-family images enable iscsid on every boot, but never install the iSCSI initiator

3 participants