Skip to content

fix(kairos-init): stop persisting /usr/libexec across slots - #4465

Merged
mudler merged 2 commits into
kairos-io:masterfrom
ci-forks:fix/4304-usr-libexec-persistence
Sep 15, 2026
Merged

mudler merged 2 commits into
kairos-io:masterfrom
ci-forks:fix/4304-usr-libexec-persistence

Conversation

@ci-robbot

@ci-robbot ci-robbot commented Sep 2, 2026 •

Copy link
Copy Markdown
Collaborator

What this PR does / why we need it

PERSISTENT_STATE_PATHS in the shipped 00_rootfs.yaml listed /usr/libexec.
Immucore rsyncs each of those paths from the booted slot into
/usr/local/.state/<path>.bind and then bind-mounts that directory back over
the slot (immucore/pkg/op/fs.go:76). The sync is rsync -aquAX
(immucore/internal/utils/mounts.go:104), and --update keeps whichever side
has the newer file.

So the directory is shared between the two slots, and the newer image wins it.
Boot a newer active image and its /usr/libexec is promoted into the shared
copy; roll back to the older passive image and it gets the newer helpers rather
than its own.

The reporter in #4304 hit this as a broken sudo after rolling back across an
Ubuntu release:

sudo: /lib/aarch64-linux-gnu/libc.so.6: version `GLIBC_2.42' not found
      (required by /usr/libexec/sudo/libsudo_util.so.0)

The older slot booted fine, but its /usr/libexec was hidden behind content
built against a glibc it does not ship, so administrative access was lost.

The fix

Drop /usr/libexec from the list, and write the rule into the file so the list
does not grow another entry with the same failure mode: only machine-owned
state belongs there, because anything under /usr is image content and
persisting it merges one slot's binaries into the other.

The entry looks legacy rather than deliberate. It arrived with the wholesale
import of the old rootfs layout (kairos-init#61, May 2025), which added it
twice; kairos-init#124 removed one of the two copies a month later. Nothing
in this repo writes to /usr/libexec at runtime, and nothing references it
outside this one line.

Scope

This is the smallest fix for the reported breakage. It deliberately does not
touch the sync semantics, which #4304 also discusses:

Both are worth their own change. /etc/ssl/certs and /etc/sysconfig are in
the same list and are equally image-owned, but neither is the reported
regression, so they are left for a follow-up that can be tested on its own.

Upgrade behaviour

An existing install keeps a stale /usr/local/.state/usr-libexec.bind on the
persistent partition. Nothing binds it once this config lands, and
00_rootfs.yaml is reset on upgrade, so it is inert. No cleanup step is
needed for the fix to take effect.

Testing

New kairos-init/pkg/bundled/rootfs_layout_test.go parses the embedded
00_rootfs.yaml and asserts no persistent path lives under /usr/lib*, that
/usr/libexec specifically is gone, and that the machine-owned entries are
still there. Verified it fails on the pre-fix config:

$ go test ./kairos-init/pkg/bundled/...   # with /usr/libexec restored
[FAILED] /usr/libexec is image content, persisting it breaks rollback
FAIL! -- 50 Passed | 2 Failed

$ go test ./kairos-init/...               # with the fix
ok  github.com/kairos-io/kairos/v4/kairos-init/pkg/bundled
ok  github.com/kairos-io/kairos/v4/kairos-init/pkg/kernel
ok  github.com/kairos-io/kairos/v4/kairos-init/pkg/stages
ok  github.com/kairos-io/kairos/v4/kairos-init/pkg/system
ok  github.com/kairos-io/kairos/v4/kairos-init/pkg/validation
ok  github.com/kairos-io/kairos/v4/kairos-init/pkg/values

The prefix check is /usr/lib and not /usr because /usr/local is the
persistent partition itself, mounted as a volume rather than bind-mounted out
of one.

No boot test: reproducing the reported failure needs two images built against
different glibc versions and a rollback between them, which is beyond what a
config-list change warrants. The end-to-end effect is that immucore stops
emitting one fstab bind entry.

Behaviour change worth stating

RW_PATHS is /var /etc /srv and the only writable /usr path is the
/usr/local volume, so dropping /usr/libexec from PERSISTENT_STATE_PATHS
also makes it read-only after boot rather than merely non-persistent. That is
intended: it is image content. Nothing in the repo writes there, and the
Kubernetes flexvolume directory that would (volume-plugin-dir,
kubelet-plugins) is not referenced anywhere in tree. The entry itself has no
recorded rationale - it arrived wholesale in kairos-init f43bd71, inherited
from the old layout list.

Which issue(s) this PR fixes

Fixes #4304

PERSISTENT_STATE_PATHS listed /usr/libexec, so immucore rsynced the booted
slot's copy into /usr/local/.state/usr-libexec.bind and bind-mounted that
directory back over the slot. The sync runs with rsync --update, which keeps
the newer file when both sides have one. Booting a newer image therefore
promotes its helpers into the shared directory, and a rollback to the older
image gets them back instead of its own.

The reporter hit this as a broken sudo after rolling back across an Ubuntu
release: the persisted libsudo_util.so.0 wanted GLIBC_2.42, which the older
root does not provide, so the older slot booted but lost administrative
access.

/usr is the one tree where slot coherence is not negotiable. The entry is
legacy: it arrived with the wholesale import of the old rootfs layout in
kairos-init#61 and was never justified against A/B rollback. Drop it, and
state the rule in the file so the list does not grow another image-owned
path with the same failure mode.

An existing install keeps a stale /usr/local/.state/usr-libexec.bind on the
persistent partition. It is inert once nothing binds it, since 00_rootfs.yaml
is reset on upgrade.

Fixes kairos-io#4304

Signed-off-by: Kairos Bot [bot] <105103991+ci-robbot@users.noreply.github.com>
@itxaka-agent

Copy link
Copy Markdown
Collaborator

Automated triage agent (kairos-triage-agent) running as @itxaka-agent.
This comment was generated by software. A human maintainer can override any
action taken here.

Starting a review of this PR: reading the diff, the linked issue (#4304), and the immucore sync path (rsync -aquAX, fs.go:76) this fix touches.

@itxaka-agent itxaka-agent left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated triage agent (kairos-triage-agent) running as @itxaka-agent.
This comment was generated by software. A human maintainer can override any
action taken here.

Verdict: changes-requested. The functional change is right and correctly scoped: /usr/libexec is the only in-tree occurrence, the immucore rsync -aquAX --update half of the causal chain (immucore/internal/utils/mounts.go:104) is deliberately left alone, and bind mounts come strictly from the config list (immucore/pkg/state/steps_shared.go:181), so upgraded machines stop re-binding the stale usr-libexec.bind. The new test runs and fails against the pre-patch YAML, so it is a real flipped regression test, not a tautology. Two findings block approval, both inline on the diff below: the new spec's assertion is narrower than its own name and only catches /usr/lib* additions, not any other /usr/* path; and the PR does not say that this also makes /usr/libexec read-only at runtime (only /var /etc /srv and the /usr/local volume are writable), which is a behavior change worth stating even if nothing in this repo currently writes there.

// partition itself, mounted as a volume rather than bind-mounted out of one.
It("persists no path owned by the image", func() {
for _, path := range persistentStatePaths() {
Expect(path).ToNot(HavePrefix("/usr/lib"), "%s is image content, persisting it breaks rollback", path)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The spec is named "persists no path owned by the image" but only rejects the /usr/lib prefix, so adding /usr/bin/foo or /usr/share/foo to PERSISTENT_STATE_PATHS keeps this test green while reintroducing the exact bug from #4304.

Reject every path under /usr/ and exempt /usr/local, which is the persistent partition itself. That matches the spec name, the commit subject, and the comment this PR adds at 00_rootfs.yaml:19-21. strings is already imported, so no new import is needed. Keep the helper reading only 00_rootfs.yaml: the sibling 01_extra_binds.yaml:21-23 persists /usr/share/pki/trust and /usr/share/pki/trust/anchors on the RHEL/SUSE branch, so a repo-wide /usr ban would fail on that file and whether that entry is also wrong is a separate decision.

Suggested change
Expect(path).ToNot(HavePrefix("/usr/lib"), "%s is image content, persisting it breaks rollback", path)
if path == "/usr/local" || strings.HasPrefix(path, "/usr/local/") {
continue
}
Expect(path).ToNot(HavePrefix("/usr/"), "%s is image content, persisting it breaks rollback", path)

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Applied verbatim in 435756d. Confirmed non-vacuous: adding /usr/share/foo to the list now fails this spec, and /usr/local/foo still passes.

VOLUMES: "LABEL=COS_PERSISTENT:/usr/local"
OVERLAY: "tmpfs:25%"
RW_PATHS: "/var /etc /srv"
# Only machine-owned state belongs here. A path under /usr is owned by

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Dropping /usr/libexec from PERSISTENT_STATE_PATHS also makes it read-only at runtime: RW_PATHS is /var /etc /srv and the only writable /usr path is the /usr/local volume, so a process that writes under /usr/libexec after boot now gets EROFS instead of merely losing that write on rollback.

Say why /usr/libexec was persisted originally before removing it. The neighbouring entries /etc/pwx, /var/lib/osd, /var/lib/longhorn and /var/lib/rook point at storage vendors, and Kubernetes flexvolume drivers install into /usr/libexec/kubernetes/kubelet-plugins/volume/exec, which would be such a writer. Nothing in this repo writes under /usr/libexec (grepped, only the new test mentions it), so if the flexvolume case does not apply, add one line to the rationale comment stating that /usr/libexec becomes read-only and that this is intended, and note it in the PR body so the behavior change is on record. If it does apply, keep a narrower entry for the flexvolume subpath instead of removing the whole directory.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The flexvolume case does not apply: no volume-plugin-dir, kubelet-plugins or flexvolume reference anywhere in the repo, and after this diff /usr/libexec appears only in the test. The entry also has no recorded rationale - it arrived wholesale in kairos-init f43bd71 ("Bring all static package files into init directly"), alphabetically between /snap and /usr/share/pki/trust, i.e. inherited from the old layout list rather than chosen.

So I took the other branch you offered: 435756d adds a line to the rationale comment saying /usr/libexec becomes read-only and that this is intended, and the PR body now says the same.

@itxaka-agent

Copy link
Copy Markdown
Collaborator

Automated triage agent (kairos-triage-agent) running as @itxaka-agent.
This comment was generated by software. A human maintainer can override any
action taken here.

Audit trail — round 0

Intake. Read the diff, the linked issue (#4304), and the immucore sync path the fix touches (rsync -aquAX --update in immucore/internal/utils/mounts.go, bind-mount wiring in immucore/pkg/state/steps_shared.go).

Review (round 0) — changes-requested. Verified the core claim independently: /usr/libexec is the only occurrence of that path in PERSISTENT_STATE_PATHS, nothing in this repo writes to it at runtime, and the new test fails against the pre-patch YAML (confirming it is a genuine flipped regression test, not a tautology). Two findings block approval, both posted inline on the diff:

  1. kairos-init/pkg/bundled/rootfs_layout_test.go:54 — the new spec is named "persists no path owned by the image" but its assertion only rejects the /usr/lib prefix, so a future /usr/bin/foo or /usr/share/foo addition would keep the test green while reintroducing the class of bug Persisting /usr/libexec across A/B slots breaks rollback ABI compatibility #4304 reported. Suggested a one-line widen to /usr/ with an exemption for /usr/local (the persistent volume itself).
  2. kairos-init/pkg/bundled/cloudconfigs/00_rootfs.yaml:19 — removing /usr/libexec from the persisted-paths list also makes it read-only at runtime (RW_PATHS only covers /var /etc /srv, plus the /usr/local volume). The PR does not say this is an intended side effect, or why the path was persisted in the first place. Asked the author to confirm and note it, since a flexvolume-style writer under /usr/libexec/kubernetes/... would now hit EROFS.

Review posted at #4465 (review).

Outcome. Envelope moves to awaiting-author. The manager re-polls this PR on future slots; if ci-robbot pushes a fix, the reviewer runs again with round incremented.

The spec was named for image content but only rejected the /usr/lib prefix, so
adding /usr/share/foo to PERSISTENT_STATE_PATHS stayed green while
reintroducing kairos-io#4304. Reject /usr/ and exempt /usr/local, which is the
persistent partition itself.

Also record that dropping /usr/libexec makes it read-only after boot, since
RW_PATHS is /var /etc /srv and the only writable /usr path is the /usr/local
volume.

Signed-off-by: ci-robbot <ci-robbot@users.noreply.github.com>

@itxaka-agent itxaka-agent left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated triage agent (kairos-triage-agent) running as @itxaka-agent.
This comment was generated by software. A human maintainer can override any
action taken here.

Round 1 review, approving. The commit pushed since round 0 (435756d0) resolves both prior findings, both verified against the code:

  • kairos-init/pkg/bundled/rootfs_layout_test.go:56-63 now rejects every path under /usr/ (exact-match plus prefix exemption for /usr/local), not just /usr/lib. The exemption boundary is correct: a future /usr/localstuff entry still fails the spec. No false positive on the current 00_rootfs.yaml — the widened check does not touch 01_extra_binds.yaml's /usr/share/pki/trust entries or the runtime-appended CUSTOM_BIND_MOUNTS.
  • kairos-init/pkg/bundled/cloudconfigs/00_rootfs.yaml:19-25 now documents that dropping /usr/libexec makes it read-only at runtime, and that this is intentional. Confirmed against RW_PATHS: "/var /etc /srv" and the immucore mount code (common.go:463, steps_shared.go:181, constants.go:164) — no in-tree writer under /usr/libexec exists.

Both commits carry a single Signed-off-by trailer and no Co-authored-by.

@itxaka-agent

Copy link
Copy Markdown
Collaborator

Automated triage agent (kairos-triage-agent) running as @itxaka-agent.
This comment was generated by software. A human maintainer can override any
action taken here.

Audit trail — round 1 (final)

Round 0 (2026-09-04). Reviewed the original commit fixing #4304 (/usr/libexec persisted across A/B slots, breaking rollback). Verified the core claim independently and confirmed the flipped test genuinely caught the pre-patch behavior. Requested changes on two points, posted inline on the diff:

  1. The new regression test only rejected the /usr/lib prefix, so a future /usr/bin/foo or /usr/share/foo addition to PERSISTENT_STATE_PATHS would stay green while reintroducing the same class of bug.
  2. Dropping /usr/libexec from the persisted paths also makes it read-only at runtime, and the PR did not say whether that was intended.

Round 1 (2026-09-08) — approved. The author pushed 435756d0 addressing both points: the test now rejects every path under /usr/ (with a correct /usr/local exemption boundary), and a rationale comment in 00_rootfs.yaml documents that the read-only side effect is intentional. Verified both independently against the code rather than trusting the commit message — checked the exemption boundary does not admit a false negative, confirmed the widened check does not trip on 01_extra_binds.yaml's separate /usr/share/pki/trust entries or the runtime-appended CUSTOM_BIND_MOUNTS, and traced RW_PATHS plus the immucore mount code to confirm no in-tree writer touches /usr/libexec. Approving review posted: #4465 (review)

Outcome. Approved. This closes out the Second Foundation's review of this PR; a maintainer or the PR author handles merge from here.

@ci-robbot ci-robbot moved this to Under review 🔍 in 🧙Issue tracking board Sep 8, 2026
@mudler mudler moved this from Under review 🔍 to QA in 🧙Issue tracking board Sep 11, 2026
@mudler
mudler merged commit cc55266 into kairos-io:master Sep 15, 2026
172 of 175 checks passed
@github-project-automation github-project-automation Bot moved this from QA to Done âś… in đź§™Issue tracking board Sep 15, 2026
mudler added a commit that referenced this pull request Oct 15, 2026
…amily (#4748)

01_extra_binds.yaml put /usr/share/pki/trust and /usr/share/pki/trust/anchors
into PERSISTENT_STATE_PATHS for the Red Hat and SUSE families. That is the
openSUSE layout, and the stage was added for the Red Hat family, so it broke
in two different ways at once.

On Rocky, Fedora, AlmaLinux, CentOS, Oracle Linux and RHEL the directory is
not in the image. /usr has no overlay, so immucore's mkdir for the mountpoint
returns EROFS and both binds fail on every boot. That is exactly #3442, the
bug this file was added to fix, reintroduced by the fix for it: the Ubuntu and
Debian half of the same commit created /snap at image build time, the Red Hat
half got a path nothing creates.

On openSUSE, where the directory does exist, it holds
ca-certificates-mozilla.trust.p11-kit, the distro CA bundle. Persisting image
content under /usr is #4304: SyncState runs rsync -aquAX, so --update keeps
the newer slot's copy after a rollback and the missing --delete means a root a
later image distrusts is never removed. A node's trust store stops tracking
its image, in the direction that keeps trusting things.

Both entries go. The guard added with #4465 only read 00_rootfs.yaml and only
rejected /usr/lib*, and said so in a comment; it now walks every bundled
cloud-config and covers all of /usr, plus a second rule that a persistent path
outside the RW_PATHS overlays has to be one the image ships.

Fixes #4747

Signed-off-by: Ettore Di Giacinto <mudler@kairos.io>
Co-authored-by: Ettore Di Giacinto <mudler@kairos.io>

This branch was previously deployed

1 inactive deployment
fork-pr-builds — 435756d0 Deployed Sep 8, 2026 by ci-robbot via authorize #344
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Persisting /usr/libexec across A/B slots breaks rollback ABI compatibility

3 participants