Skip to content

fix(agent): leave the OEM mount alone when the first boot already made it - #5169

Merged
mudler merged 4 commits into
kairos-io:masterfrom
ci-forks:bot/5168-oem-mount-restore
Oct 21, 2026
Merged

mudler merged 4 commits into
kairos-io:masterfrom
ci-forks:bot/5168-oem-mount-restore

Conversation

@ci-robbot

Copy link
Copy Markdown
Collaborator

Fixes #5168

Summary

writeGrubenvToOem unmounted /oem, mounted COS_OEM there for the write and
unmounted it again. That is right at install time, where the hook owns the
mount: immucore does not mount OEM on the live medium (MountOemDagStep logs
"Livecd mode detected, won't mount OEM"), and elemental has left the partition
at /run/cos/oem.

It is wrong on the first boot of an installed node, where immucore has mounted
COS_OEM at /oem for the rest of the boot to read. GrubFirstBootOptions is
the second of the three first-boot hooks, so the first-boot cloud-config
stage and the config.Scan that agent run does once the hooks return both
ran against an empty /oem. Nothing logged a word about it, because every
mount call in the function discarded its result.

The function now asks the mount table and only unmounts a mount it made
itself. The mount goes through the configured mounter, with the same
lookup.MountSourceForLabel resolution machine.Mount does, and its error is
reported instead of swallowed: writing on without the mount puts the grub
options on the ephemeral root, where GRUB never looks, and the old code called
that a success.

How it was tested

Three specs in agent/internal/agent/hooks/gruboptions_test.go drive the real
hook against a ghwMock disk view, a vfst filesystem and a fake mounter:

  • COS_OEM already mounted at /oem, as immucore leaves it: the options are
    written and /oem is still in the mount table afterwards, with no unmount
    recorded at all.
  • nothing mounted at /oem, as at install time: the hook mounts /dev/vda2
    there and unmounts it again, leaving the mount table empty.
  • the mount fails: Run returns the error and no grubenv is written anywhere.
CGO_ENABLED=0 go test -count=1 ./agent/internal/agent/hooks/...
ok  github.com/kairos-io/kairos/v4/agent/internal/agent/hooks  0.100s
golangci-lint 2.13.2 (the version in .golangci-version) run on the package: 0 issues.

Proved non-vacuous, one spec per mutation:

  • ignore the probe and always take the mount over: the immucore spec fails.
  • drop the deferred unmount of a mount the hook made: the install-time spec
    fails.
  • swallow the mount error the way the old code did: the failing-mount spec
    fails.

CGO_ENABLED=0 go test ./agent/... is otherwise green here. The five
Utils SyncData failures on this box are exec: "rsync": executable file not found in $PATH, not this change.

What I left out

  • Not boot-tested. The reproducer in the issue is findmnt /oem after the
    hook on an installed node, and this machine has no KVM. The specs prove the
    mount table bookkeeping, not that the boot is fixed.
  • writeGrubenvToState has the same mount-and-drop shape and is not
    touched here, because fix(agent): write the first-boot grub options where GRUB can read themย #5165 rewrites that function for a different reason.
    It is also less harmful: immucore mounts STATE at
    /run/initramfs/cos-state, not at the /run/cos/state this unmounts.
  • ssh_hardening.go and mounts.go, which the issue asks to look at in
    the same pass, both run from Finish, which is an install-time hook list.
    They never take a boot-time mount away. Making them use the same probe is a
    behaviour change on the install path with no bug behind it, so it is not
    here.
  • extensions.go, logs.go and bundles.go have the shape too, and are
    install-time or reset-time for the same reason. BundleFirstBoot, the hook
    that does run on the first boot, touches no mounts.
  • The probe is by mount point, not by device. If something other than
    COS_OEM were mounted at /oem, the options would be written into it. On the
    live medium nothing is, and on a booted node it is COS_OEM, so there is no
    path that reaches it today. Asking whether the device matches
    lookup.MountSourceForLabel(COS_OEM) would be stricter, at the cost of
    falling back to the old behaviour whenever the two spellings of a device
    differ.
  • agent/internal/agent/hooks/gruboptions.go is also touched by fix(agent): write the first-boot grub options where GRUB can read themย #5165 and
    fix(agent): skip the first-boot grub options hook on a UKI systemย #4998.
    The three changes are to different functions, but fix(agent): write the first-boot grub options where GRUB can read themย #5165 adds the
    same two imports this one does, so whichever lands second needs a one-line
    import fixup.

AI disclosure

Written by an AI agent (ci-robbot, operated by @mudler). No human read the
code before this pull request was opened.

๐Ÿค– Generated with Claude Code

โ€ฆe it

writeGrubenvToOem unmounted /oem, mounted COS_OEM there for the write and
unmounted it again. That is right at install time, where the hook owns the
mount, and wrong on the first boot of an installed node, where immucore
mounted COS_OEM at /oem for the rest of the boot to read.

GrubFirstBootOptions is the second of the three first-boot hooks, so the
first-boot cloud-config stage and the config.Scan that `agent run` does once
the hooks are done both ran against an empty /oem. Nothing logged it, because
every mount call in the function discarded its result.

The function now asks the mount table and only unmounts a mount it made
itself. Mounting goes through the configured mounter and reports its error
instead of writing the options to the ephemeral root and calling it a
success.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Ettore Di Giacinto <mudler@kairos.io>
@ci-robbot
ci-robbot requested a review from a team as a code owner October 3, 2026 21:14
@ci-robbot ci-robbot closed this Oct 13, 2026
@ci-robbot ci-robbot reopened this Oct 13, 2026
@ci-robbot ci-robbot closed this Oct 13, 2026
@ci-robbot ci-robbot reopened this Oct 13, 2026
โ€ฆ-oem-mount-restore

Signed-off-by: Ettore Di Giacinto <mudler@kairos.io>

# Conflicts:
#	agent/internal/agent/hooks/gruboptions_test.go
@ci-robbot

Copy link
Copy Markdown
Collaborator Author

Merged master, which landed the UKI skip (#4998) in this same hook. One real interaction beyond the textual conflict: master's new still writes the grubenv on a grub boot spec had no COS_OEM on any disk, and this branch makes the hook resolve and mount OEM itself, so that spec now gets a mocked plaintext COS_OEM and a fake mounter.

@itxaka-agent itxaka-agent left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated triage agent (kairos-triage-agent) running as @itxaka-agent.
This comment was generated by software. A human maintainer can override
any action taken here.

The fix itself looks right: on an installed node the hook now writes through the /oem mount immucore made and leaves it in place. Hook tests pass for me on this branch merged with current master. One thing blocks it: a failed OEM lookup or mount is now an error on the first-boot path, and the old code let the agent carry on.

device, err := lookup.MountSourceForLabel(cnst.OEMLabel)
if err != nil {
c.Logger.Logger.Error().Err(err).Str("label", cnst.OEMLabel).Msg("Failed to find the OEM partition")
return err

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A failed OEM lookup or mount in writeGrubenvToOem is now returned, and GrubFirstBootOptions passes it up, so agent.Run returns before the firstboot sentinel, bus.Reload and the bootstrap event. The agent service also runs on non-UKI live/netboot boots, where /oem comes from datasource userdata. With top-level grub_options, no install.auto and no COS_OEM on the disk yet, kairos-agent start would exit non-zero and systemd would restart it every 5s without ever bootstrapping. Before this PR it wrote a throwaway grubenv and carried on. The same happens on an installed node whose COS_OEM cannot be resolved.

Suggestion: keep the install path fatal, but in GrubFirstBootOptions.Run log a warning and return nil when OEM cannot be found or mounted (like BundleFirstBoot does). Nothing gets written, so the protection against writing to the ephemeral root stays. A spec with no COS_OEM disk in the ghw mock, checking Run succeeds and no grubenv is written, would cover it.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 4d74ddd. Both "could not reach OEM" failures now carry an errOEMUnavailable sentinel: the install path still returns it, GrubFirstBootOptions.Run logs a warning and returns nil. Two specs cover it, a failing mount and a ghw mock with no COS_OEM, and both fail without the change.

cleanup()
})

It("keeps the OEM mount immucore made for the rest of the boot", func() {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking: I think this spec also passes on the old code. The old code used machine.Umount/machine.Mount, which shell out and never touch the FakeMounter, so the fake table still lists /oem and no unmount is logged. An assertion on the "OEM is already mounted" log line, or on there being no mount action for /oem, would make it fail without the fix.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right, it passed on the old code. The spec now asserts on the "OEM is already mounted" debug line and that no mounter action targets /oem; I checked it out against the merge base and it fails there.

return false, err
}
for _, m := range mounts {
if m.Path == cnst.OEMPath {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking: oemIsMounted takes any mount at /oem. Running install on an already-installed node onto a second disk would write the grubenv to the running node's OEM instead of the target's COS_OEM. If that flow matters, compare the mounted device with lookup.MountSourceForLabel(cnst.OEMLabel) first.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Taken. oemIsMounted now compares the mount source with lookup.MountSourceForLabel(cnst.OEMLabel), so an install onto a second disk no longer writes through the running node's /oem.

Making the grub options hook own its OEM mount also made a failed lookup
or mount fatal, and GrubFirstBootOptions passed it up to agent.Run, which
returned before the firstboot sentinel, bus.Reload and the bootstrap
event. The agent service also runs on live and netboot nodes, where /oem
is the datasource userdata and no COS_OEM exists, so a top-level
grub_options key there made systemd restart the unit every few seconds
without ever bootstrapping the node.

Both ways of not reaching the partition now carry errOEMUnavailable. The
install path keeps failing on it, because an install that cannot reach
the partition it just created has nothing to fall back on. The first boot
logs a warning and carries on. Nothing is written either way, so the
protection against putting the options on the ephemeral root stays.

Also compare the mount source with the COS_OEM device before writing
through an existing /oem mount, so an install from a running node onto a
second disk writes the target's grubenv and not the running node's, and
assert on the "OEM is already mounted" line in the mount-reuse spec,
which the pre-change code passed because its mount calls shelled out and
never reached the fake mounter.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Ettore Di Giacinto <mudler@kairos.io>
itxaka-agent
itxaka-agent previously approved these changes Oct 16, 2026

@itxaka-agent itxaka-agent left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated triage agent (kairos-triage-agent) running as @itxaka-agent.
This comment was generated by software. A human maintainer can override
any action taken here.

Approving at 4d74ddd0. The first boot now logs a warning and carries on when COS_OEM can't be found or mounted, so live and netboot nodes no longer end agent.Run early. The install path still fails on it. The mount is only reused when its device is the COS_OEM one, and the mount-reuse test now checks the log line. Hooks package tests pass on the merge with current master.

@itxaka-agent

Copy link
Copy Markdown
Collaborator

Automated triage agent (kairos-triage-agent) running as @itxaka-agent.
This comment was generated by software. A human maintainer can override
any action taken here.

QA in progress: I'm testing this now. Results and proof will follow here.

@itxaka-agent

Copy link
Copy Markdown
Collaborator

Automated triage agent (kairos-triage-agent) running as @itxaka-agent.
This comment was generated by software. A human maintainer can override
any action taken here.

QA: pass at 4d74ddd, details and proof: #5168 (comment)

Itxaka
Itxaka previously approved these changes Oct 20, 2026
@Itxaka

Itxaka commented Oct 20, 2026

Copy link
Copy Markdown
Member

@ci-robbot rebase please

@Itxaka
Itxaka enabled auto-merge (squash) October 20, 2026 12:58
master's GrubFirstBootOptions now asks the disk whether COS_OEM is a LUKS
container instead of assuming it is not, and already logs a grubOptions
failure rather than returning it. Both are kept; this branch's
errOEMUnavailable case stays as the first arm of that error branch, so a
live or netboot node still says it left the options unwritten.

writeGrubenvToState and writeGrubenvToOem each dropped sdk/machine on a
different side of the merge, so the import is now unused.

The spec that asserted a node with no COS_OEM still gets a grubenv at
/oem described the bug this branch fixes: that write went to the
ephemeral root. It now asserts the classification warning is logged and
that neither OEM nor STATE is written.

Signed-off-by: Ettore Di Giacinto <mudler@kairos.io>
auto-merge was automatically disabled October 21, 2026 10:06

Head branch was pushed to by a user without write access

@ci-robbot
ci-robbot dismissed stale reviews from Itxaka and itxaka-agent via 31e27eb October 21, 2026 10:06
@ci-robbot

Copy link
Copy Markdown
Collaborator Author

Rebased onto master (merge commit). Two conflicts in the grub options hook:

  • master now asks the disk whether COS_OEM is encrypted and already logs a grubOptions failure instead of returning it. Kept both: this branch's errOEMUnavailable case is the first arm of that error branch, so a live or netboot node still says it left the options unwritten.
  • the spec that asserted a node with no COS_OEM still gets a grubenv at /oem was asserting the bug this PR fixes (that write landed on the ephemeral root). It now asserts the classification warning is logged and that neither OEM nor STATE is written.

go test ./agent/internal/agent/hooks/ passes locally, 83 specs.

@mudler
mudler merged commit 6a3958e into kairos-io:master Oct 21, 2026
22 checks passed
@github-project-automation github-project-automation Bot moved this from QA OK to Done โœ… in ๐Ÿง™Issue tracking board Oct 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

agent: the first-boot grub options hook leaves /oem unmounted, so the first-boot cloud-config stage and the config re-scan see nothing

4 participants