Repository navigation
fix(kairos-init): stop persisting /usr/libexec across slots - #4465
Conversation
PERSISTENT_STATE_PATHS listed /usr/libexec, so immucore rsynced the booted slot's copy into /usr/local/.state/usr-libexec.bind and bind-mounted that directory back over the slot. The sync runs with rsync --update, which keeps the newer file when both sides have one. Booting a newer image therefore promotes its helpers into the shared directory, and a rollback to the older image gets them back instead of its own. The reporter hit this as a broken sudo after rolling back across an Ubuntu release: the persisted libsudo_util.so.0 wanted GLIBC_2.42, which the older root does not provide, so the older slot booted but lost administrative access. /usr is the one tree where slot coherence is not negotiable. The entry is legacy: it arrived with the wholesale import of the old rootfs layout in kairos-init#61 and was never justified against A/B rollback. Drop it, and state the rule in the file so the list does not grow another image-owned path with the same failure mode. An existing install keeps a stale /usr/local/.state/usr-libexec.bind on the persistent partition. It is inert once nothing binds it, since 00_rootfs.yaml is reset on upgrade. Fixes kairos-io#4304 Signed-off-by: Kairos Bot [bot] <105103991+ci-robbot@users.noreply.github.com>
Starting a review of this PR: reading the diff, the linked issue (#4304), and the immucore sync path ( |
itxaka-agent
left a comment
There was a problem hiding this comment.
Automated triage agent (
kairos-triage-agent) running as@itxaka-agent.
This comment was generated by software. A human maintainer can override any
action taken here.
Verdict: changes-requested. The functional change is right and correctly scoped: /usr/libexec is the only in-tree occurrence, the immucore rsync -aquAX --update half of the causal chain (immucore/internal/utils/mounts.go:104) is deliberately left alone, and bind mounts come strictly from the config list (immucore/pkg/state/steps_shared.go:181), so upgraded machines stop re-binding the stale usr-libexec.bind. The new test runs and fails against the pre-patch YAML, so it is a real flipped regression test, not a tautology. Two findings block approval, both inline on the diff below: the new spec's assertion is narrower than its own name and only catches /usr/lib* additions, not any other /usr/* path; and the PR does not say that this also makes /usr/libexec read-only at runtime (only /var /etc /srv and the /usr/local volume are writable), which is a behavior change worth stating even if nothing in this repo currently writes there.
| // partition itself, mounted as a volume rather than bind-mounted out of one. | ||
| It("persists no path owned by the image", func() { | ||
| for _, path := range persistentStatePaths() { | ||
| Expect(path).ToNot(HavePrefix("/usr/lib"), "%s is image content, persisting it breaks rollback", path) |
There was a problem hiding this comment.
The spec is named "persists no path owned by the image" but only rejects the /usr/lib prefix, so adding /usr/bin/foo or /usr/share/foo to PERSISTENT_STATE_PATHS keeps this test green while reintroducing the exact bug from #4304.
Reject every path under /usr/ and exempt /usr/local, which is the persistent partition itself. That matches the spec name, the commit subject, and the comment this PR adds at 00_rootfs.yaml:19-21. strings is already imported, so no new import is needed. Keep the helper reading only 00_rootfs.yaml: the sibling 01_extra_binds.yaml:21-23 persists /usr/share/pki/trust and /usr/share/pki/trust/anchors on the RHEL/SUSE branch, so a repo-wide /usr ban would fail on that file and whether that entry is also wrong is a separate decision.
| Expect(path).ToNot(HavePrefix("/usr/lib"), "%s is image content, persisting it breaks rollback", path) | |
| if path == "/usr/local" || strings.HasPrefix(path, "/usr/local/") { | |
| continue | |
| } | |
| Expect(path).ToNot(HavePrefix("/usr/"), "%s is image content, persisting it breaks rollback", path) |
There was a problem hiding this comment.
Applied verbatim in 435756d. Confirmed non-vacuous: adding /usr/share/foo to the list now fails this spec, and /usr/local/foo still passes.
| VOLUMES: "LABEL=COS_PERSISTENT:/usr/local" | ||
| OVERLAY: "tmpfs:25%" | ||
| RW_PATHS: "/var /etc /srv" | ||
| # Only machine-owned state belongs here. A path under /usr is owned by |
There was a problem hiding this comment.
Dropping /usr/libexec from PERSISTENT_STATE_PATHS also makes it read-only at runtime: RW_PATHS is /var /etc /srv and the only writable /usr path is the /usr/local volume, so a process that writes under /usr/libexec after boot now gets EROFS instead of merely losing that write on rollback.
Say why /usr/libexec was persisted originally before removing it. The neighbouring entries /etc/pwx, /var/lib/osd, /var/lib/longhorn and /var/lib/rook point at storage vendors, and Kubernetes flexvolume drivers install into /usr/libexec/kubernetes/kubelet-plugins/volume/exec, which would be such a writer. Nothing in this repo writes under /usr/libexec (grepped, only the new test mentions it), so if the flexvolume case does not apply, add one line to the rationale comment stating that /usr/libexec becomes read-only and that this is intended, and note it in the PR body so the behavior change is on record. If it does apply, keep a narrower entry for the flexvolume subpath instead of removing the whole directory.
There was a problem hiding this comment.
The flexvolume case does not apply: no volume-plugin-dir, kubelet-plugins or flexvolume reference anywhere in the repo, and after this diff /usr/libexec appears only in the test. The entry also has no recorded rationale - it arrived wholesale in kairos-init f43bd71 ("Bring all static package files into init directly"), alphabetically between /snap and /usr/share/pki/trust, i.e. inherited from the old layout list rather than chosen.
So I took the other branch you offered: 435756d adds a line to the rationale comment saying /usr/libexec becomes read-only and that this is intended, and the PR body now says the same.
Audit trail — round 0Intake. Read the diff, the linked issue (#4304), and the immucore sync path the fix touches ( Review (round 0) — changes-requested. Verified the core claim independently:
Review posted at #4465 (review). Outcome. Envelope moves to |
The spec was named for image content but only rejected the /usr/lib prefix, so adding /usr/share/foo to PERSISTENT_STATE_PATHS stayed green while reintroducing kairos-io#4304. Reject /usr/ and exempt /usr/local, which is the persistent partition itself. Also record that dropping /usr/libexec makes it read-only after boot, since RW_PATHS is /var /etc /srv and the only writable /usr path is the /usr/local volume. Signed-off-by: ci-robbot <ci-robbot@users.noreply.github.com>
itxaka-agent
left a comment
There was a problem hiding this comment.
Automated triage agent (
kairos-triage-agent) running as@itxaka-agent.
This comment was generated by software. A human maintainer can override any
action taken here.
Round 1 review, approving. The commit pushed since round 0 (435756d0) resolves both prior findings, both verified against the code:
kairos-init/pkg/bundled/rootfs_layout_test.go:56-63now rejects every path under/usr/(exact-match plus prefix exemption for/usr/local), not just/usr/lib. The exemption boundary is correct: a future/usr/localstuffentry still fails the spec. No false positive on the current00_rootfs.yaml— the widened check does not touch01_extra_binds.yaml's/usr/share/pki/trustentries or the runtime-appendedCUSTOM_BIND_MOUNTS.kairos-init/pkg/bundled/cloudconfigs/00_rootfs.yaml:19-25now documents that dropping/usr/libexecmakes it read-only at runtime, and that this is intentional. Confirmed againstRW_PATHS: "/var /etc /srv"and the immucore mount code (common.go:463,steps_shared.go:181,constants.go:164) — no in-tree writer under/usr/libexecexists.
Both commits carry a single Signed-off-by trailer and no Co-authored-by.
Audit trail — round 1 (final)Round 0 (2026-09-04). Reviewed the original commit fixing #4304 (
Round 1 (2026-09-08) — approved. The author pushed Outcome. Approved. This closes out the Second Foundation's review of this PR; a maintainer or the PR author handles merge from here. |
…amily (#4748) 01_extra_binds.yaml put /usr/share/pki/trust and /usr/share/pki/trust/anchors into PERSISTENT_STATE_PATHS for the Red Hat and SUSE families. That is the openSUSE layout, and the stage was added for the Red Hat family, so it broke in two different ways at once. On Rocky, Fedora, AlmaLinux, CentOS, Oracle Linux and RHEL the directory is not in the image. /usr has no overlay, so immucore's mkdir for the mountpoint returns EROFS and both binds fail on every boot. That is exactly #3442, the bug this file was added to fix, reintroduced by the fix for it: the Ubuntu and Debian half of the same commit created /snap at image build time, the Red Hat half got a path nothing creates. On openSUSE, where the directory does exist, it holds ca-certificates-mozilla.trust.p11-kit, the distro CA bundle. Persisting image content under /usr is #4304: SyncState runs rsync -aquAX, so --update keeps the newer slot's copy after a rollback and the missing --delete means a root a later image distrusts is never removed. A node's trust store stops tracking its image, in the direction that keeps trusting things. Both entries go. The guard added with #4465 only read 00_rootfs.yaml and only rejected /usr/lib*, and said so in a comment; it now walks every bundled cloud-config and covers all of /usr, plus a second rule that a persistent path outside the RW_PATHS overlays has to be one the image ships. Fixes #4747 Signed-off-by: Ettore Di Giacinto <mudler@kairos.io> Co-authored-by: Ettore Di Giacinto <mudler@kairos.io>
What this PR does / why we need it
PERSISTENT_STATE_PATHSin the shipped00_rootfs.yamllisted/usr/libexec.Immucore rsyncs each of those paths from the booted slot into
/usr/local/.state/<path>.bindand then bind-mounts that directory back overthe slot (
immucore/pkg/op/fs.go:76). The sync isrsync -aquAX(
immucore/internal/utils/mounts.go:104), and--updatekeeps whichever sidehas the newer file.
So the directory is shared between the two slots, and the newer image wins it.
Boot a newer active image and its
/usr/libexecis promoted into the sharedcopy; roll back to the older passive image and it gets the newer helpers rather
than its own.
The reporter in #4304 hit this as a broken
sudoafter rolling back across anUbuntu release:
The older slot booted fine, but its
/usr/libexecwas hidden behind contentbuilt against a glibc it does not ship, so administrative access was lost.
The fix
Drop
/usr/libexecfrom the list, and write the rule into the file so the listdoes not grow another entry with the same failure mode: only machine-owned
state belongs there, because anything under
/usris image content andpersisting it merges one slot's binaries into the other.
The entry looks legacy rather than deliberate. It arrived with the wholesale
import of the old rootfs layout (
kairos-init#61, May 2025), which added ittwice;
kairos-init#124removed one of the two copies a month later. Nothingin this repo writes to
/usr/libexecat runtime, and nothing references itoutside this one line.
Scope
This is the smallest fix for the reported breakage. It deliberately does not
touch the sync semantics, which #4304 also discusses:
--updateis there on purpose (immucore#525) to protect user-modifiedstate, so it cannot simply be dropped for
/etc/sshor/etc/systemd.--delete, so removed files outlive the image that wrotethem. I hit that same class of bug from a different direction in fix(immucore): move stale unit symlinks out of the systemd load path #4452.
Both are worth their own change.
/etc/ssl/certsand/etc/sysconfigare inthe same list and are equally image-owned, but neither is the reported
regression, so they are left for a follow-up that can be tested on its own.
Upgrade behaviour
An existing install keeps a stale
/usr/local/.state/usr-libexec.bindon thepersistent partition. Nothing binds it once this config lands, and
00_rootfs.yamlis reset on upgrade, so it is inert. No cleanup step isneeded for the fix to take effect.
Testing
New
kairos-init/pkg/bundled/rootfs_layout_test.goparses the embedded00_rootfs.yamland asserts no persistent path lives under/usr/lib*, that/usr/libexecspecifically is gone, and that the machine-owned entries arestill there. Verified it fails on the pre-fix config:
The prefix check is
/usr/liband not/usrbecause/usr/localis thepersistent partition itself, mounted as a volume rather than bind-mounted out
of one.
No boot test: reproducing the reported failure needs two images built against
different glibc versions and a rollback between them, which is beyond what a
config-list change warrants. The end-to-end effect is that immucore stops
emitting one fstab bind entry.
Behaviour change worth stating
RW_PATHSis/var /etc /srvand the only writable/usrpath is the/usr/localvolume, so dropping/usr/libexecfromPERSISTENT_STATE_PATHSalso makes it read-only after boot rather than merely non-persistent. That is
intended: it is image content. Nothing in the repo writes there, and the
Kubernetes flexvolume directory that would (
volume-plugin-dir,kubelet-plugins) is not referenced anywhere in tree. The entry itself has norecorded rationale - it arrived wholesale in kairos-init
f43bd71, inheritedfrom the old layout list.
Which issue(s) this PR fixes
Fixes #4304