Skip to content

fix(agent): resolve the persistent label before copying the install logs - #4687

Merged
mudler merged 1 commit into
kairos-io:masterfrom
ci-forks:fix/4685-copylogs-bylabel
Sep 29, 2026
Merged

mudler merged 1 commit into
kairos-io:masterfrom
ci-forks:fix/4685-copylogs-bylabel

Conversation

@ci-robbot

Copy link
Copy Markdown
Collaborator

Fixes #4685

CopyLogs handed /dev/disk/by-label/COS_PERSISTENT straight to mount(2):

c.Syscall.Mount(filepath.Join("/dev/disk/by-label", constants.PersistentLabel), constants.PersistentDir, "ext4", 0, "")

On an encrypted install that path is ambiguous. The raw crypto_LUKS container and its unlocked mapper both advertise COS_PERSISTENT, so the symlink points at whichever device udev settled last. The hook runs from Finish immediately after Encrypt, while the mapper is still open, so it sits in that window every time. When the symlink lands on the raw side the mount fails, the hook warns and returns nil, and the install and reset logs are silently dropped on exactly the nodes that are hardest to debug after the fact.

This is the same defect #4403 fixed for immucore's mount step and for machine.Mount. The line right above this one already mounts OEM through the fixed helper, so this call site was just missed in that sweep.

The fix routes the label through lookup.MountSourceForLabel, which returns the mapper path for a LUKS-wrapped partition and the plain partition path otherwise. c.Syscall.Mount is kept so the hook stays mockable.

Three specs in logs_test.go drive the real hook against a GhwMock disk. All three fail on master:

  • encrypted layout (vda5 = crypto_LUKS carrying the label) must mount /dev/mapper/vda5, never the by-label path
  • unencrypted layout must mount /dev/vda5, never the by-label path
  • no COS_PERSISTENT at all must warn and return nil, never mount the by-label path

That last one is worth calling out: today the hook mounts the by-label path unconditionally, so on a machine where the label does not resolve it was relying on mount(2) to fail rather than checking.

The two hardening suggestions in the issue are not in scope here. The requested <= current grow error comes from yip's layout plugin, not this repo. Hard-failing the boot when /usr/local does not mount is a behaviour change that wants a maintainer call.

CopyLogs handed /dev/disk/by-label/COS_PERSISTENT straight to mount(2).
On an encrypted install that path is ambiguous: the raw crypto_LUKS
container and its unlocked mapper both advertise the label, so udev's
last writer decides where the symlink points. The hook runs immediately
after Encrypt, while the mapper is still open, so it hits the ambiguous
window every time. When the symlink lands on the raw side the mount
fails, the hook warns and returns nil, and the install and reset logs
are dropped on exactly the nodes where they are hardest to reproduce.

Resolve the label through lookup.MountSourceForLabel, which returns the
mapper for a LUKS-wrapped partition and the plain partition otherwise.
The line above already mounted OEM through the fixed machine.Mount
helper, so this call site was simply missed when kairos-io#4403 swept the rest.

Fixes kairos-io#4685

Signed-off-by: Ettore Di Giacinto <mudler@kairos.io>
@ci-robbot
ci-robbot requested a review from a team as a code owner September 17, 2026 00:11
@ci-robbot ci-robbot moved this to Under review πŸ” in πŸ§™Issue tracking board Sep 17, 2026
@mudler mudler moved this from Under review πŸ” to QA in πŸ§™Issue tracking board Sep 18, 2026
@ci-robbot

Copy link
Copy Markdown
Collaborator Author

Picking this up for QA.

@ci-robbot

Copy link
Copy Markdown
Collaborator Author

QA Result: PASS

Check Result
Environment hadron standard ISO v0.5.1 / v4.3.0-85-g96ef1c97d, QEMU q35 + KVM, 30 GiB virtio disk, swtpm TPM 2.0, kcrypt "Local TPM NV passphrase". A/B of the same make kairos multi-call binary bind-mounted over /usr/bin/kairos: PR b3e9325b2 vs base 73706deff.
Reproduction Real encrypted install, then the reported ambiguous label shape, then kairos-agent reset --unattended from the recovery entry with each binary in turn, off one shared disk snapshot.
Expected CopyLogs mounts the plaintext filesystem (the mapper on an encrypted node), so the install/reset logs reach the persistent partition whichever device won the by-label symlink.
Actual Base drops the reset logs (could not mount persistent, persistent left with only lost+found). PR copies them. Unencrypted install and post-reset boot unaffected.
Proofs Full logs, transcript and mutation check: https://gist.github.com/ci-robbot/aa93a1b185fda11b20c2e7923cba3c3e

What was tested

  1. Install path, encrypted, PR binary. manual-install with install.encrypted_partitions: [COS_PERSISTENT]. CopyLogs ran right after Encrypt and landed agent.log (33997 bytes) in .state/var-log.bind/kairos/ on persistent.

    Worth recording, because it changes where this fix actually bites: a fresh post-Encrypted persistent boot flakes: LUKS container and inner ext4 share LABEL=COS_PERSISTENTΒ #4403 install is not ambiguous. The container comes out labelled COS_PERSISTENT_LUKS and only the mapper carries COS_PERSISTENT, so by-label/COS_PERSISTENT has a single claimant and base would have worked too. The PR description's "it sits in that window every time" holds for the label shape in the report, not for a freshly installed node on current master.

  2. The reported shape. cryptsetup config /dev/vda5 --label COS_PERSISTENT on the installed disk reproduces a pre-Encrypted persistent boot flakes: LUKS container and inner ext4 share LABEL=COS_PERSISTENTΒ #4403 node: both /dev/vda5 (crypto_LUKS) and /dev/mapper/vda5 (ext4) claim disk/by-label/COS_PERSISTENT. On the next boot udev resolved that symlink to the raw container, with no udev rule and no tampering:

    $ ls /etc/udev/rules.d/ | grep -i qa || echo NO_QA_UDEV_RULE_PRESENT
    NO_QA_UDEV_RULE_PRESENT
    $ ls -la /dev/disk/by-label/COS_PERSISTENT
    COS_PERSISTENT -> ../../vda5
    $ mount -t ext4 /dev/disk/by-label/COS_PERSISTENT /mnt/p
    mount: fsconfig() failed: /dev/vda5: Can't open blockdev.   (rc=32)
    $ mount -t ext4 /dev/mapper/vda5 /mnt/p
    MAPPER_MOUNT_OK
    
  3. Reset A/B, from the recovery entry. Same backing image, same TPM state, same by-label -> /dev/vda5 in both arms. A timestamp marker was dropped in /var/log/ before each run.

    base 73706deff PR b3e9325b2
    CopyLogs WRN could not mount persistent error="device or resource busy" INF Starting rsync... DBG Logs copied to persistent partition
    persistent after reset only lost+found, no .state/ at all .state/var-log.bind/kairos/agent.log (30223 bytes)
    marker file absent .state/var-log.bind/qa-reset-marker-PR present

    On the base arm the reset log existed only in the recovery tmpfs and was gone after reboot, which is the failure mode the issue describes: no evidence on exactly the node that needs it.

    reset itself is unaffected either way, it formats through /dev/mapper/vda5 already (mkfs.ext4 -L COS_PERSISTENT /dev/mapper/vda5). Only the log hook was still on the ambiguous path.

  4. No regressions. The PR arm's reset node reboots into active, persistent mounts at /usr/local through the mapper, and the preserved agent.log is still there and being appended to. A separate unencrypted install with the PR binary also copies its logs (vda5 ext4 COS_PERSISTENT, agent.log 29602 bytes), so MountSourceForLabel returns the plain partition and not a mapper path that does not exist.

  5. Specs. go test ./agent/internal/agent/hooks/... green on the PR. Reverting only logs.go to 73706def and keeping the new specs turns all three red, so they are not vacuous.

Summary: the fix does what it claims. With both devices carrying COS_PERSISTENT and udev resolving to the raw LUKS container, the reset logs are lost on base and preserved on the PR, and neither the encrypted install path, the unencrypted install path, nor the post-reset boot changed. Note for the reviewer: the failure needs the pre-#4403 label shape, so this is a fix for nodes installed before #4403 (which is what the issue reports), not for nodes freshly installed on master.

QA performed by an AI agent, unattended. No human has reviewed this run.

This branch was previously deployed

1 inactive deployment
fork-pr-builds β€” b3e9325b Deployed Sep 17, 2026 by ci-robbot via authorize #567
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

immucore: encrypted partitions β€” mounting via /dev/disk/by-label is non-deterministic when the raw LUKS device and the mapper share a label

2 participants