Repository navigation
Description
Kairos version
- Kairos v4.1.2
- kairos-agent v2.30.0
- kairos-init v0.15.0
- immucore v0.18.0
- Non-UKI GRUB boot
- Upgrade crossed Ubuntu releases
Describe the bug
Kairos includes /usr/libexec in the default PERSISTENT_STATE_PATHS. Immucore copies this directory into /usr/local/.state/usr-libexec.bind and bind-mounts it over the selected slot's own /usr/libexec.
The copy uses rsync -aquAX. Because --update prevents an older source file from replacing a newer destination file, booting a newer active image updates the persisted helpers, but subsequently rolling back to an older passive image does not restore that image's older helpers.
The result is a mixed system: executable code from the newer image under /usr/libexec is combined with libraries from the older root filesystem.
In the observed rollback, this made sudo unusable:
sudo: /lib/aarch64-linux-gnu/libc.so.6: version `GLIBC_2.42' not found
(required by /usr/libexec/sudo/libsudo_util.so.0)
The newer persisted libsudo_util.so.0 required a glibc symbol that was not provided by the rolled-back image. The older slot itself was bootable, but normal administrative access was lost because its /usr/libexec had been hidden by incompatible persisted content.
To reproduce
- Install and boot a Kairos image based on an older distribution release.
- Upgrade to an image based on a newer distribution release with a newer glibc and sudo build.
- Boot the newer active image successfully so its
/usr/libexeccontents are synchronized into persistent state. - Boot the older passive image through fallback or manual slot selection.
- Run
sudo.
The passive image boots with the persisted /usr/libexec from the newer image, and sudo fails during dynamic linking.
The persisted directory can be observed at:
/usr/local/.state/usr-libexec.bind
Expected behavior
An A/B rollback should expose a coherent filesystem from the selected slot. Image-owned executables and shared objects under /usr/libexec should come from that slot and remain compatible with its libraries.
/usr/libexec should not be persisted across slots by default. If persistence is required for some contents, it should be scoped to the image or slot rather than merged globally.
Source
kairos-init includes /usr/libexec in the default persistent paths:
Immucore synchronizes the selected slot into the persistent directory and then bind-mounts that directory over the slot path:
The synchronization uses rsync -aquAX, preserving newer destination files during rollback:
The --update behavior was introduced in:
The same default persistence and synchronization behavior remains in current releases.
Metadata
Metadata
Assignees
Labels
Type
Projects
- StatusShow more project fieldsDone ✅
Activity
wyvernzora commentedon Aug 12, 2026
One additional consequence of the current synchronization is that
rsync -aquAXdoes not use--delete. In addition to--updateretaining newer destination copies during rollback, files that exist in the persistent destination but have been removed from the selected image are not removed by this synchronization. Persisted image-owned directories can therefore accumulate stale files across upgrades and rollbacks, further weakening slot coherence.For
/usr/libexec, removing it from the default persistent paths still appears to be the cleanest fix. If the generic persistence synchronization remains applicable to image-owned directories, its destination deletion semantics may also need consideration.ci-robbot commentedon Sep 2, 2026
Still reproducible on current
master(c80ae704). The source links in the report point at the now-archivedkairos-initandimmucorerepos, so for anyone landing here after the monorepo migration, the same code is at:kairos-init/pkg/bundled/cloudconfigs/00_rootfs.yaml:37—/usr/libexecis still inPERSISTENT_STATE_PATHS.immucore/pkg/op/fs.go:76,101— still builds<state>/usr-libexec.bindand callsSyncStatebefore bind-mounting it over the slot path.immucore/internal/utils/mounts.go:104— stillrsync -aquAX, no--delete.The analysis holds. Two independent defects share that one rsync invocation, and it is worth keeping them apart because they need different fixes:
--updateinverts on rollback. Newer-in-destination wins, so the persisted/usr/libexecfrom the newer slot survives into the older one. That is thelibsudo_util.so.0/GLIBC_2.42failure.--updatewas added deliberately (immucore#525) to protect user-modified state, so it cannot simply be dropped for the paths where that protection is the point (/etc/ssh,/etc/systemd).--delete, as @wyvernzora notes: the sync only ever adds, so anything an earlier image wrote outlives that image. I hit the same class of bug from a different direction in fix(immucore): move stale unit symlinks out of the systemd load path #4452, where asshd.servicealias symlink written by an Ubuntu image kept shadowing a Hadron image's own unit after upgrade. Same root cause, different path.Agreed that removing
/usr/libexecfrom the default list is the cleanest fix for the reported breakage, and it is a much smaller blast radius than changing the sync semantics for every persistent path at once. The list mixes two kinds of entry that happen to share one mechanism: genuinely machine-owned state (/home,/root,/opt,/var/lib/*) and image-owned directories that are only there because something writes into them at runtime./usr/libexecis squarely the second kind, and/usris the one tree where slot coherence is not negotiable.Worth a look while someone is in there:
/etc/ssl/certsand/etc/sysconfigare in the same list and are equally image-owned, and #4414 proposes adding/etc/ssl/privatealongside. Whatever is decided for/usr/libexecprobably wants to be a stated rule rather than a one-off removal, otherwise the list keeps growing entries with this failure mode.ci-robbot commentedon Sep 2, 2026
Picked this up. PR: #4465.
Scoped to what the report asks for:
/usr/libexecis out of the defaultPERSISTENT_STATE_PATHS, with the rule stated in the file and a test that parses the shipped config and rejects any persistent path under/usr/lib*.Left out on purpose, since neither is the reported regression and both want their own change:
--updatesemantics, which exist deliberately (immucore#525) and cannot be dropped for the paths where protecting user-modified state is the point;--delete@wyvernzora raised;/etc/ssl/certsand/etc/sysconfig, which are image-owned by the same argument.@wyvernzora if you have a slot pair from the original failure still around, a rollback on an image built from this branch would be a useful confirmation.
fix(kairos-init): reject every /usr path, not just /usr/lib
fix(kairos-init): stop persisting /usr/libexec across slots (kairos-i…
fix(cloudconfigs): stop persisting the SUSE CA bundle path on every f…
docs(examples): show how to make /usr/libexec writable but ephemeral (#…