Skip to content

Persisting /usr/libexec across A/B slots breaks rollback ABI compatibility #4304

Closed

Description

@wyvernzora

Kairos version

  • Kairos v4.1.2
  • kairos-agent v2.30.0
  • kairos-init v0.15.0
  • immucore v0.18.0
  • Non-UKI GRUB boot
  • Upgrade crossed Ubuntu releases

Describe the bug

Kairos includes /usr/libexec in the default PERSISTENT_STATE_PATHS. Immucore copies this directory into /usr/local/.state/usr-libexec.bind and bind-mounts it over the selected slot's own /usr/libexec.

The copy uses rsync -aquAX. Because --update prevents an older source file from replacing a newer destination file, booting a newer active image updates the persisted helpers, but subsequently rolling back to an older passive image does not restore that image's older helpers.

The result is a mixed system: executable code from the newer image under /usr/libexec is combined with libraries from the older root filesystem.

In the observed rollback, this made sudo unusable:

sudo: /lib/aarch64-linux-gnu/libc.so.6: version `GLIBC_2.42' not found
      (required by /usr/libexec/sudo/libsudo_util.so.0)

The newer persisted libsudo_util.so.0 required a glibc symbol that was not provided by the rolled-back image. The older slot itself was bootable, but normal administrative access was lost because its /usr/libexec had been hidden by incompatible persisted content.

To reproduce

  1. Install and boot a Kairos image based on an older distribution release.
  2. Upgrade to an image based on a newer distribution release with a newer glibc and sudo build.
  3. Boot the newer active image successfully so its /usr/libexec contents are synchronized into persistent state.
  4. Boot the older passive image through fallback or manual slot selection.
  5. Run sudo.

The passive image boots with the persisted /usr/libexec from the newer image, and sudo fails during dynamic linking.

The persisted directory can be observed at:

/usr/local/.state/usr-libexec.bind

Expected behavior

An A/B rollback should expose a coherent filesystem from the selected slot. Image-owned executables and shared objects under /usr/libexec should come from that slot and remain compatible with its libraries.

/usr/libexec should not be persisted across slots by default. If persistence is required for some contents, it should be scoped to the image or slot rather than merged globally.

Source

kairos-init includes /usr/libexec in the default persistent paths:

https://github.com/kairos-io/kairos-init/blob/e090b91ffa93747d34422f795820d82813bfd660/pkg/bundled/cloudconfigs/00_rootfs.yaml#L19-L56

Immucore synchronizes the selected slot into the persistent directory and then bind-mounts that directory over the slot path:

https://github.com/kairos-io/immucore/blob/03ce6004d48d228697e7c76e152f56f163324b08/pkg/op/fs.go#L70-L102

The synchronization uses rsync -aquAX, preserving newer destination files during rollback:

https://github.com/kairos-io/immucore/blob/03ce6004d48d228697e7c76e152f56f163324b08/internal/utils/mounts.go#L95-L105

The --update behavior was introduced in:

kairos-io/immucore#525

The same default persistence and synchronization behavior remains in current releases.

Activity

  1. wyvernzora commented on Aug 12, 2026

    @wyvernzora
    Author

    One additional consequence of the current synchronization is that rsync -aquAX does not use --delete. In addition to --update retaining newer destination copies during rollback, files that exist in the persistent destination but have been removed from the selected image are not removed by this synchronization. Persisted image-owned directories can therefore accumulate stale files across upgrades and rollbacks, further weakening slot coherence.

    For /usr/libexec, removing it from the default persistent paths still appears to be the cleanest fix. If the generic persistence synchronization remains applicable to image-owned directories, its destination deletion semantics may also need consideration.

  2. moved this to In Progress 🏃 in 🧙Issue tracking boardon Aug 12, 2026
  3. moved this from In Progress 🏃 to Todo 🖊 in 🧙Issue tracking boardon Aug 12, 2026
  4. moved this from Todo 🖊 to In Progress 🏃 in 🧙Issue tracking boardon Aug 24, 2026
  5. moved this from In Progress 🏃 to Todo 🖊 in 🧙Issue tracking boardon Aug 24, 2026
  6. ci-robbot commented on Sep 2, 2026

    @ci-robbot
    Collaborator

    Still reproducible on current master (c80ae704). The source links in the report point at the now-archived kairos-init and immucore repos, so for anyone landing here after the monorepo migration, the same code is at:

    • kairos-init/pkg/bundled/cloudconfigs/00_rootfs.yaml:37 — /usr/libexec is still in PERSISTENT_STATE_PATHS.
    • immucore/pkg/op/fs.go:76,101 — still builds <state>/usr-libexec.bind and calls SyncState before bind-mounting it over the slot path.
    • immucore/internal/utils/mounts.go:104 — still rsync -aquAX, no --delete.

    The analysis holds. Two independent defects share that one rsync invocation, and it is worth keeping them apart because they need different fixes:

    1. --update inverts on rollback. Newer-in-destination wins, so the persisted /usr/libexec from the newer slot survives into the older one. That is the libsudo_util.so.0 / GLIBC_2.42 failure. --update was added deliberately (immucore#525) to protect user-modified state, so it cannot simply be dropped for the paths where that protection is the point (/etc/ssh, /etc/systemd).
    2. No --delete, as @wyvernzora notes: the sync only ever adds, so anything an earlier image wrote outlives that image. I hit the same class of bug from a different direction in fix(immucore): move stale unit symlinks out of the systemd load path #4452, where a sshd.service alias symlink written by an Ubuntu image kept shadowing a Hadron image's own unit after upgrade. Same root cause, different path.

    Agreed that removing /usr/libexec from the default list is the cleanest fix for the reported breakage, and it is a much smaller blast radius than changing the sync semantics for every persistent path at once. The list mixes two kinds of entry that happen to share one mechanism: genuinely machine-owned state (/home, /root, /opt, /var/lib/*) and image-owned directories that are only there because something writes into them at runtime. /usr/libexec is squarely the second kind, and /usr is the one tree where slot coherence is not negotiable.

    Worth a look while someone is in there: /etc/ssl/certs and /etc/sysconfig are in the same list and are equally image-owned, and #4414 proposes adding /etc/ssl/private alongside. Whatever is decided for /usr/libexec probably wants to be a stated rule rather than a one-off removal, otherwise the list keeps growing entries with this failure mode.

  7. ci-robbot commented on Sep 2, 2026

    @ci-robbot
    Collaborator

    Picked this up. PR: #4465.

    Scoped to what the report asks for: /usr/libexec is out of the default PERSISTENT_STATE_PATHS, with the rule stated in the file and a test that parses the shipped config and rejects any persistent path under /usr/lib*.

    Left out on purpose, since neither is the reported regression and both want their own change:

    • the --update semantics, which exist deliberately (immucore#525) and cannot be dropped for the paths where protecting user-modified state is the point;
    • the missing --delete @wyvernzora raised;
    • /etc/ssl/certs and /etc/sysconfig, which are image-owned by the same argument.

    @wyvernzora if you have a slot pair from the original failure still around, a rollback on an image built from this branch would be a useful confirmation.

  8. moved this from Todo 🖊 to Under review 🔍 in 🧙Issue tracking boardon Sep 8, 2026
  9. moved this from Under review 🔍 to Done ✅ in 🧙Issue tracking boardon Sep 15, 2026
  10. added a commit that references this issue on Sep 15, 2026
    cc55266
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions