Skip to content

fix(ci): keep /dev/kvm writable across the udev re-trigger - #4562

Merged
mudler merged 1 commit into
kairos-io:masterfrom
ci-forks:fix/kvm-perms-udev-race
Sep 9, 2026
Merged

mudler merged 1 commit into
kairos-io:masterfrom
ci-forks:fix/kvm-perms-udev-race

Conversation

@ci-robbot

Copy link
Copy Markdown
Collaborator

Fixes #4561

The problem

provider-decentralized-k8s aborted on master in
run 34251428760:

VM Aborted.
stderr: Could not access KVM kernel module: Permission denied
qemu-system-x86_64: failed to initialize kvm: Permission denied

The Enable KVM group perms step in the same job was green. It runs under
bash -e, so setfacl did apply. Permission denied, not No such file or directory, so the node existed throughout. The ACL was set and then lost:

17:38:26.7  Setting up qemu-system-common
17:38:27.2  Setting up libvirt-daemon-system
17:38:30.8  apt finishes, setfacl runs, exits 0
17:38:32.0  step ends green
17:38:45.0  Could not access KVM kernel module: Permission denied

Installing the qemu/libvirt stack triggers udev. udevadm trigger is
asynchronous: it queues the events and returns, and udevd applies the packaged
/dev/kvm ownership (root:kvm, 0660) whenever it reaches them. Nothing in
the step waited for that queue, so setfacl and udevd raced, and the ACL lost.

Only provider-decentralized-k8s runs on a GitHub-hosted runner
(reusable-qemu-test.yaml:133),
so it is the only cell that installs libvirt and qemu from scratch on every
run, and the only one that can open this window. On the self-hosted fast
label apt-get install is a no-op and nothing triggers udev.

The change

  sudo apt-get install -y libvirt-clients ... ovmf curl
+ echo 'KERNEL=="kvm", GROUP="kvm", MODE="0666", OPTIONS+="static_node=kvm"' | sudo tee /etc/udev/rules.d/99-kvm4all.rules
+ sudo udevadm control --reload-rules || true
+ sudo udevadm trigger --name-match=kvm || true
+ sudo udevadm settle || true
- sudo setfacl -m u:runner:rwx /dev/kvm
+ sudo setfacl -m u:runner:rwx /dev/kvm || true
+ test -w /dev/kvm

A rule in /etc/udev/rules.d/ outranks the packaged one, so the mode holds
however late udevd gets to the event. udevadm settle closes the window the
postinst triggers leave open.

setfacl stays as a fallback for a runner where udevadm is not usable,
which is why the udev commands and the ACL are all || true. The step's
verdict now comes from test -w /dev/kvm instead, which is the condition that
actually matters, and it fails in two seconds rather than letting the suite die
minutes later on a spec that has nothing to do with permissions.

The same step in _uki-test.yaml gets the same treatment. It runs on the
self-hosted kvm label where this does not currently bite, but the step is a
copy of the same code and the new check passes there for the same reason it
passes today.

Verification

Both files parse (gopkg.in/yaml.v3), and the generated shell block is clean
under bash -n. The race itself only reproduces on a GitHub-hosted runner
doing a cold libvirt install, so CI on this PR is the test.

This is distinct from #4557 and its PR #4558, which are about the panic
swallowing the logs. This one is about why the VM aborted in the first place.

provider-decentralized-k8s aborted on master with "Could not access KVM
kernel module: Permission denied" while its own Enable KVM group perms
step was green. The step runs under bash -e, so setfacl did apply. The
ACL was set and then lost.

Installing the qemu/libvirt stack triggers udev. udevadm trigger is
asynchronous: it queues the events and returns, and udevd applies the
packaged root:kvm 0660 whenever it reaches them. Nothing waited for that
queue, so setfacl and udevd raced and the ACL lost.

Write a rule in /etc/udev/rules.d/ that outranks the packaged one, so
the mode holds however late udevd processes the event, then settle to
close the window the postinst triggers leave open. setfacl stays as a
fallback for a runner where udevadm is not usable, hence the || true.

Finish with test -w /dev/kvm. Today the step that owns the device passes
and the suite dies minutes later on an unrelated spec; this names the
real problem in two seconds.

Only provider-decentralized-k8s runs on a GitHub-hosted runner
(reusable-qemu-test.yaml:133), so it is the only cell that installs
libvirt and qemu from scratch and the only one that can hit this. The
same step in _uki-test.yaml gets the same treatment.

Fixes kairos-io#4561

Signed-off-by: ci-robbot <ci-robbot@users.noreply.github.com>
@ci-robbot
ci-robbot requested a review from a team as a code owner September 9, 2026 03:12
@ci-robbot ci-robbot moved this to Under review πŸ” in πŸ§™Issue tracking board Sep 9, 2026
@mudler
mudler enabled auto-merge (squash) September 9, 2026 10:11
@ci-robbot ci-robbot moved this from Under review πŸ” to QA in πŸ§™Issue tracking board Sep 9, 2026
@ci-robbot

Copy link
Copy Markdown
Collaborator Author

Picking this up for QA.

@ci-robbot

Copy link
Copy Markdown
Collaborator Author

QA Result: PASS

Check Result
Environment PR head 97d4f839, replayed on a real systemd-udevd host (Linux 6.8.0-139-generic, Ubuntu, /dev/kvm present, packaged rule 50-udev-default.rules:114 = root:kvm 0660)
Reproduction Ran master's step (setfacl only) and then let a udev event reach /dev/kvm; then ran the PR's step verbatim and replayed the same udev event on top of it. Also parsed both workflow files and bash -n'd the generated run: block.
Expected After the step, /dev/kvm is writable by the job user, and stays writable when udev processes the events queued by the qemu/libvirt postinst.
Actual Matches. On master the named ACL is destroyed by the udev event; with the rule installed the ACL is destroyed just the same but MODE=0666 holds, and a user outside the kvm group can still open the device. test -w /dev/kvm exits 0.
Proofs gist: full session log + write-up

Summary: the fix does what it claims, verified end to end against a real udev, not just by reading the diff. One correction to the PR body: CI on this PR is not a test of it.


The failure mode is real, and the diagnosis is slightly off in a way that does not change the fix.

The ACL does not survive a udev event on the device:

$ sudo setfacl -m u:nobody:rwx /dev/kvm     # stands in for u:runner
exit=0                                       <-- the step is GREEN here
ls: crw-rwxrw-+ root kvm   acl: user:nobody:rwx

$ sudo udevadm trigger --name-match=kvm && sudo udevadm settle
ls: crw-rw-rw-+ root kvm   acl: (none)

What wipes it is the uaccess builtin, not the mode reset: 70-uaccess.rules:48 tags
KERNEL=="kvm" and 73-seat-late.rules:16 runs builtin:uaccess, which rewrites the
device ACL from scratch. So the node is not "re-created", it is re-processed, and
re-processing throws the ACL away. Same window, same outcome, and the chosen fix is
the right one for it: a mode is not an ACL, so uaccess cannot take it away.

The rule does outrank the packaged one. udevadm test /devices/virtual/misc/kvm
with the file in place:

Reading rules file: /etc/udev/rules.d/99-kvm4all.rules
kvm: /usr/lib/udev/rules.d/50-udev-default.rules:114 MODE 0660
kvm: /etc/udev/rules.d/99-kvm4all.rules:1 MODE 0666
kvm: Setting permissions /dev/kvm, uid=0, gid=994, mode=0666

Without the file the same command ends at mode=0660.

The late event replayed against the fix:

before trigger: crw-rwxrw-+ root kvm   acl=1
$ sudo udevadm trigger --action=add --name-match=kvm && sudo udevadm settle
after  trigger: crw-rw-rw-+ root kvm   acl=0
probe(nobody, not in the kvm group): WRITABLE

The ACL goes, KVM stays. That is the property the PR is buying.


Finding, not a blocker: "CI on this PR is the test" does not hold.

The only run on the head SHA is 34306218077, event pull_request_target. That event
resolves uses: ./.github/workflows/... against the base branch, so the reusable
workflow that ran came from master. The provider-decentralized-k8s job log
(102429921350) prints the step body it executed:

##[group]Run sudo apt-get update
sudo apt-get update
sudo apt-get install -y libvirt-clients ... ovmf curl
sudo setfacl -m u:runner:rwx /dev/kvm        <-- master's one-liner
shell: /usr/bin/bash -e {0}

No udevadm, no 99-kvm4all.rules, no test -w. The green run means master's step
happened to win the race that time, nothing more. The change first executes for real
after it merges, which is worth knowing if it needs a follow-up.

Two smaller notes, neither worth holding the PR for:

  • On the self-hosted fast and kvm labels the step now writes a file into
    /etc/udev/rules.d/ on the host, and that file outlives the job. Harmless, and
    arguably what you want there, but it is persistent runner state that master's step
    did not create.
  • sudo tee echoes the rule into the log. Cosmetic.

Verified with a real udev, and the host was restored to its baseline afterwards.

This branch was previously deployed

1 inactive deployment
fork-pr-builds β€” 97d4f839 Deployed Sep 9, 2026 by ci-robbot via authorize #374
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Archived in project

Development

Successfully merging this pull request may close these issues.

CI: provider-decentralized-k8s aborts with "Could not access KVM kernel module: Permission denied" although Enable KVM group perms is green

2 participants