Repository navigation
fix(ci): keep /dev/kvm writable across the udev re-trigger - #4562
Conversation
provider-decentralized-k8s aborted on master with "Could not access KVM kernel module: Permission denied" while its own Enable KVM group perms step was green. The step runs under bash -e, so setfacl did apply. The ACL was set and then lost. Installing the qemu/libvirt stack triggers udev. udevadm trigger is asynchronous: it queues the events and returns, and udevd applies the packaged root:kvm 0660 whenever it reaches them. Nothing waited for that queue, so setfacl and udevd raced and the ACL lost. Write a rule in /etc/udev/rules.d/ that outranks the packaged one, so the mode holds however late udevd processes the event, then settle to close the window the postinst triggers leave open. setfacl stays as a fallback for a runner where udevadm is not usable, hence the || true. Finish with test -w /dev/kvm. Today the step that owns the device passes and the suite dies minutes later on an unrelated spec; this names the real problem in two seconds. Only provider-decentralized-k8s runs on a GitHub-hosted runner (reusable-qemu-test.yaml:133), so it is the only cell that installs libvirt and qemu from scratch and the only one that can hit this. The same step in _uki-test.yaml gets the same treatment. Fixes kairos-io#4561 Signed-off-by: ci-robbot <ci-robbot@users.noreply.github.com>
|
Picking this up for QA. |
QA Result: PASS
Summary: the fix does what it claims, verified end to end against a real udev, not just by reading the diff. One correction to the PR body: CI on this PR is not a test of it. The failure mode is real, and the diagnosis is slightly off in a way that does not change the fix. The ACL does not survive a udev event on the device: What wipes it is the The rule does outrank the packaged one. Without the file the same command ends at The late event replayed against the fix: The ACL goes, KVM stays. That is the property the PR is buying. Finding, not a blocker: "CI on this PR is the test" does not hold. The only run on the head SHA is No Two smaller notes, neither worth holding the PR for:
Verified with a real udev, and the host was restored to its baseline afterwards. |
Fixes #4561
The problem
provider-decentralized-k8saborted onmasterinrun 34251428760:
The
Enable KVM group permsstep in the same job was green. It runs underbash -e, sosetfacldid apply.Permission denied, notNo such file or directory, so the node existed throughout. The ACL was set and then lost:Installing the qemu/libvirt stack triggers udev.
udevadm triggerisasynchronous: it queues the events and returns, and udevd applies the packaged
/dev/kvmownership (root:kvm,0660) whenever it reaches them. Nothing inthe step waited for that queue, so
setfacland udevd raced, and the ACL lost.Only
provider-decentralized-k8sruns on a GitHub-hosted runner(
reusable-qemu-test.yaml:133),so it is the only cell that installs libvirt and qemu from scratch on every
run, and the only one that can open this window. On the self-hosted
fastlabel
apt-get installis a no-op and nothing triggers udev.The change
A rule in
/etc/udev/rules.d/outranks the packaged one, so the mode holdshowever late udevd gets to the event.
udevadm settlecloses the window thepostinst triggers leave open.
setfaclstays as a fallback for a runner whereudevadmis not usable,which is why the udev commands and the ACL are all
|| true. The step'sverdict now comes from
test -w /dev/kvminstead, which is the condition thatactually matters, and it fails in two seconds rather than letting the suite die
minutes later on a spec that has nothing to do with permissions.
The same step in
_uki-test.yamlgets the same treatment. It runs on theself-hosted
kvmlabel where this does not currently bite, but the step is acopy of the same code and the new check passes there for the same reason it
passes today.
Verification
Both files parse (
gopkg.in/yaml.v3), and the generated shell block is cleanunder
bash -n. The race itself only reproduces on a GitHub-hosted runnerdoing a cold libvirt install, so CI on this PR is the test.
This is distinct from #4557 and its PR #4558, which are about the panic
swallowing the logs. This one is about why the VM aborted in the first place.