Repository navigation
fix(agent): leave the OEM mount alone when the first boot already made it - #5169
Conversation
โฆe it writeGrubenvToOem unmounted /oem, mounted COS_OEM there for the write and unmounted it again. That is right at install time, where the hook owns the mount, and wrong on the first boot of an installed node, where immucore mounted COS_OEM at /oem for the rest of the boot to read. GrubFirstBootOptions is the second of the three first-boot hooks, so the first-boot cloud-config stage and the config.Scan that `agent run` does once the hooks are done both ran against an empty /oem. Nothing logged it, because every mount call in the function discarded its result. The function now asks the mount table and only unmounts a mount it made itself. Mounting goes through the configured mounter and reports its error instead of writing the options to the ephemeral root and calling it a success. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Ettore Di Giacinto <mudler@kairos.io>
โฆ-oem-mount-restore Signed-off-by: Ettore Di Giacinto <mudler@kairos.io> # Conflicts: # agent/internal/agent/hooks/gruboptions_test.go
|
Merged master, which landed the UKI skip (#4998) in this same hook. One real interaction beyond the textual conflict: master's new |
itxaka-agent
left a comment
There was a problem hiding this comment.
Automated triage agent (
kairos-triage-agent) running as@itxaka-agent.
This comment was generated by software. A human maintainer can override
any action taken here.
The fix itself looks right: on an installed node the hook now writes through the /oem mount immucore made and leaves it in place. Hook tests pass for me on this branch merged with current master. One thing blocks it: a failed OEM lookup or mount is now an error on the first-boot path, and the old code let the agent carry on.
| device, err := lookup.MountSourceForLabel(cnst.OEMLabel) | ||
| if err != nil { | ||
| c.Logger.Logger.Error().Err(err).Str("label", cnst.OEMLabel).Msg("Failed to find the OEM partition") | ||
| return err |
There was a problem hiding this comment.
A failed OEM lookup or mount in writeGrubenvToOem is now returned, and GrubFirstBootOptions passes it up, so agent.Run returns before the firstboot sentinel, bus.Reload and the bootstrap event. The agent service also runs on non-UKI live/netboot boots, where /oem comes from datasource userdata. With top-level grub_options, no install.auto and no COS_OEM on the disk yet, kairos-agent start would exit non-zero and systemd would restart it every 5s without ever bootstrapping. Before this PR it wrote a throwaway grubenv and carried on. The same happens on an installed node whose COS_OEM cannot be resolved.
Suggestion: keep the install path fatal, but in GrubFirstBootOptions.Run log a warning and return nil when OEM cannot be found or mounted (like BundleFirstBoot does). Nothing gets written, so the protection against writing to the ephemeral root stays. A spec with no COS_OEM disk in the ghw mock, checking Run succeeds and no grubenv is written, would cover it.
There was a problem hiding this comment.
Done in 4d74ddd. Both "could not reach OEM" failures now carry an errOEMUnavailable sentinel: the install path still returns it, GrubFirstBootOptions.Run logs a warning and returns nil. Two specs cover it, a failing mount and a ghw mock with no COS_OEM, and both fail without the change.
| cleanup() | ||
| }) | ||
|
|
||
| It("keeps the OEM mount immucore made for the rest of the boot", func() { |
There was a problem hiding this comment.
Non-blocking: I think this spec also passes on the old code. The old code used machine.Umount/machine.Mount, which shell out and never touch the FakeMounter, so the fake table still lists /oem and no unmount is logged. An assertion on the "OEM is already mounted" log line, or on there being no mount action for /oem, would make it fail without the fix.
There was a problem hiding this comment.
Right, it passed on the old code. The spec now asserts on the "OEM is already mounted" debug line and that no mounter action targets /oem; I checked it out against the merge base and it fails there.
| return false, err | ||
| } | ||
| for _, m := range mounts { | ||
| if m.Path == cnst.OEMPath { |
There was a problem hiding this comment.
Non-blocking: oemIsMounted takes any mount at /oem. Running install on an already-installed node onto a second disk would write the grubenv to the running node's OEM instead of the target's COS_OEM. If that flow matters, compare the mounted device with lookup.MountSourceForLabel(cnst.OEMLabel) first.
There was a problem hiding this comment.
Taken. oemIsMounted now compares the mount source with lookup.MountSourceForLabel(cnst.OEMLabel), so an install onto a second disk no longer writes through the running node's /oem.
Making the grub options hook own its OEM mount also made a failed lookup or mount fatal, and GrubFirstBootOptions passed it up to agent.Run, which returned before the firstboot sentinel, bus.Reload and the bootstrap event. The agent service also runs on live and netboot nodes, where /oem is the datasource userdata and no COS_OEM exists, so a top-level grub_options key there made systemd restart the unit every few seconds without ever bootstrapping the node. Both ways of not reaching the partition now carry errOEMUnavailable. The install path keeps failing on it, because an install that cannot reach the partition it just created has nothing to fall back on. The first boot logs a warning and carries on. Nothing is written either way, so the protection against putting the options on the ephemeral root stays. Also compare the mount source with the COS_OEM device before writing through an existing /oem mount, so an install from a running node onto a second disk writes the target's grubenv and not the running node's, and assert on the "OEM is already mounted" line in the mount-reuse spec, which the pre-change code passed because its mount calls shelled out and never reached the fake mounter. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Ettore Di Giacinto <mudler@kairos.io>
itxaka-agent
left a comment
There was a problem hiding this comment.
Automated triage agent (
kairos-triage-agent) running as@itxaka-agent.
This comment was generated by software. A human maintainer can override
any action taken here.
Approving at 4d74ddd0. The first boot now logs a warning and carries on when COS_OEM can't be found or mounted, so live and netboot nodes no longer end agent.Run early. The install path still fails on it. The mount is only reused when its device is the COS_OEM one, and the mount-reuse test now checks the log line. Hooks package tests pass on the merge with current master.
QA in progress: I'm testing this now. Results and proof will follow here. |
QA: pass at 4d74ddd, details and proof: #5168 (comment) |
|
@ci-robbot rebase please |
master's GrubFirstBootOptions now asks the disk whether COS_OEM is a LUKS container instead of assuming it is not, and already logs a grubOptions failure rather than returning it. Both are kept; this branch's errOEMUnavailable case stays as the first arm of that error branch, so a live or netboot node still says it left the options unwritten. writeGrubenvToState and writeGrubenvToOem each dropped sdk/machine on a different side of the merge, so the import is now unused. The spec that asserted a node with no COS_OEM still gets a grubenv at /oem described the bug this branch fixes: that write went to the ephemeral root. It now asserts the classification warning is logged and that neither OEM nor STATE is written. Signed-off-by: Ettore Di Giacinto <mudler@kairos.io>
Head branch was pushed to by a user without write access
|
Rebased onto master (merge commit). Two conflicts in the grub options hook:
|
Fixes #5168
Summary
writeGrubenvToOemunmounted/oem, mounted COS_OEM there for the write andunmounted it again. That is right at install time, where the hook owns the
mount: immucore does not mount OEM on the live medium (
MountOemDagSteplogs"Livecd mode detected, won't mount OEM"), and elemental has left the partition
at
/run/cos/oem.It is wrong on the first boot of an installed node, where immucore has mounted
COS_OEM at
/oemfor the rest of the boot to read.GrubFirstBootOptionsisthe second of the three first-boot hooks, so the
first-bootcloud-configstage and the
config.Scanthatagent rundoes once the hooks return bothran against an empty
/oem. Nothing logged a word about it, because everymount call in the function discarded its result.
The function now asks the mount table and only unmounts a mount it made
itself. The mount goes through the configured mounter, with the same
lookup.MountSourceForLabelresolutionmachine.Mountdoes, and its error isreported instead of swallowed: writing on without the mount puts the grub
options on the ephemeral root, where GRUB never looks, and the old code called
that a success.
How it was tested
Three specs in
agent/internal/agent/hooks/gruboptions_test.godrive the realhook against a
ghwMockdisk view, avfstfilesystem and a fake mounter:/oem, as immucore leaves it: the options arewritten and
/oemis still in the mount table afterwards, with no unmountrecorded at all.
/oem, as at install time: the hook mounts/dev/vda2there and unmounts it again, leaving the mount table empty.
Runreturns the error and no grubenv is written anywhere.Proved non-vacuous, one spec per mutation:
fails.
fails.
CGO_ENABLED=0 go test ./agent/...is otherwise green here. The fiveUtils SyncDatafailures on this box areexec: "rsync": executable file not found in $PATH, not this change.What I left out
findmnt /oemafter thehook on an installed node, and this machine has no KVM. The specs prove the
mount table bookkeeping, not that the boot is fixed.
writeGrubenvToStatehas the same mount-and-drop shape and is nottouched here, because fix(agent): write the first-boot grub options where GRUB can read themย #5165 rewrites that function for a different reason.
It is also less harmful: immucore mounts STATE at
/run/initramfs/cos-state, not at the/run/cos/statethis unmounts.ssh_hardening.goandmounts.go, which the issue asks to look at inthe same pass, both run from
Finish, which is an install-time hook list.They never take a boot-time mount away. Making them use the same probe is a
behaviour change on the install path with no bug behind it, so it is not
here.
extensions.go,logs.goandbundles.gohave the shape too, and areinstall-time or reset-time for the same reason.
BundleFirstBoot, the hookthat does run on the first boot, touches no mounts.
COS_OEM were mounted at
/oem, the options would be written into it. On thelive medium nothing is, and on a booted node it is COS_OEM, so there is no
path that reaches it today. Asking whether the device matches
lookup.MountSourceForLabel(COS_OEM)would be stricter, at the cost offalling back to the old behaviour whenever the two spellings of a device
differ.
agent/internal/agent/hooks/gruboptions.gois also touched by fix(agent): write the first-boot grub options where GRUB can read themย #5165 andfix(agent): skip the first-boot grub options hook on a UKI systemย #4998. The three changes are to different functions, but fix(agent): write the first-boot grub options where GRUB can read themย #5165 adds the
same two imports this one does, so whichever lands second needs a one-line
import fixup.
AI disclosure
Written by an AI agent (
ci-robbot, operated by @mudler). No human read thecode before this pull request was opened.
๐ค Generated with Claude Code