Skip to content

ci(tests): give the two multi-peer qemu legs a bridge network - #5533

Merged
mudler merged 1 commit into
kairos-io:masterfrom
ci-forks:bot/5494-bridge-network-p2p-legs
Oct 20, 2026
Merged

mudler merged 1 commit into
kairos-io:masterfrom
ci-forks:bot/5494-bridge-network-p2p-legs

Conversation

@ci-robbot

@ci-robbot ci-robbot commented Oct 19, 2026 •

Copy link
Copy Markdown
Collaborator

Fixes #5494

Summary

test-standard / provider-decentralized-k8s and
test-standard / provider-qrcode-install are the only two legs in that
matrix that need a second libp2p peer to be discovered. The first pairs
the two VMs over edgevpn, the second pairs the runner with the VM over
go-nodepair. The other three legs drive a single VM from the runner over
the forwarded SSH port.

On QEMU user-mode networking neither pair can see the other, so the
rendezvous has to go out to the public DHT and come back. That works most
of the time, which is why these two are the legs that redden
intermittently while the three single-peer legs stay green on the same run
and the same image.

reusable-qemu-test.yaml already carries the whole fix and nothing turned
it on:

  • it declares a use_bridge_network input;
  • when that input is true it starts the libvirt default network, allows
    virbr0 in the QEMU bridge helper and verifies virbr0 exists;
  • it passes USE_BRIDGE_NETWORK to the suite, and the suite adds a second
    NIC on virbr0;
  • the dependency step installs libvirt-daemon-system, libvirt-clients
    and bridge-utils on every qemu job already, so nothing new is pulled
    in.

No caller in the repository ever set the input, so none of that has run.
master.yaml and pr-full.yaml now set it for those two legs, through a
bridge key on the matrix entry so the other three are untouched.

The second change is the fail-open on the suite side. When
USE_BRIDGE_NETWORK was set and ensureBridgeNetwork failed, the failure
was logged as a warning and the VM started on user-mode networking anyway.
The spec then failed ten minutes later at its own rendezvous timeout,
naming discovery rather than the missing bridge. It is now an error.

Closing the question #5494 was parked on

The issue asked for a re-run of master at 4433f215 to tell a
rendezvous flake apart from a regression in ce57f7fc..4433f215, and I
could not trigger one. Four later master runs have answered it:

Run Date sha qrcode-install decentralized-k8s
38405470043 10-13 0145342c success 4m success 7m
38585196692 10-14 ce57f7fc success 5m success 6m
38699913175 10-15 4433f215 failure 12m failure 45m
39145797866 10-18 5ac6129c success 4m success 6m
39226114428 10-19 581c17be success 5m failure 19m

Two green runs at shas later than 4433f215 rule the merge range out. The
10-19 red has the same signature as the 10-15 one, a 600s Eventually
with a single peer for its whole length:

Waiting for nodes: have 1, need 2 (advertizing=[2f8173e41af247d69babef7196ebfa7b-kairos-3k2y] active=[12D3KooWAYT12MCEfCcXFue32VCsEXzC6BDgJsDrMDgbRwaUDPgu])
not enough nodes available, sleeping... needed: 2, available: 1

So it is the rendezvous, and it has been one all along.

How it was tested

A guard in internal/ciguard/bridge_network_test.go, next to the
permissions guard, which reads the workflow YAML and never touches the
network. Three specs, each mutation-checked one at a time against this
branch:

Mutation Spec that fails
bridge: true dropped from provider-decentralized-k8s in master.yaml passes use_bridge_network to the reusable workflow, naming the leg and the file
use_bridge_network dropped from both callers' with: that spec, plus has at least one caller that sets it, which is the state this change found
the callee stops passing USE_BRIDGE_NETWORK to the suite is an input reusable-qemu-test.yaml declares and consumes

go test ./internal/ciguard/... is green, golangci-lint run at the
version in .golangci-version reports 0 issues on that package and on the
tests module, go vet ./... in tests is clean, and actionlint is
clean on both changed workflows.

What this does not prove

I cannot run a qemu leg here, so the bridge path itself has not been
exercised end to end.

It will not run on this pull request by itself either: pr-full.yaml is
pull_request_target on labeled, and its authorize job requires the
labeller to have write access, which ci-robbot does not. So the first
real exercise is either a maintainer adding ci/full here, or the next
push to master after this merges.

If virbr0 cannot be brought up on a hosted runner, the
Setup bridge network step fails at ip link show virbr0 within
seconds, and the suite change makes the other failure mode loud as well,
so a wrong answer here costs one fast red rather than another 45-minute
timeout.

AI was used to write this. No human has read it yet.

πŸ€– Generated with Claude Code

provider-decentralized-k8s and provider-qrcode-install are the only
test-standard legs that need a second libp2p peer to be discovered: the
first pairs the two VMs over edgevpn, the second pairs the runner with
the VM over go-nodepair. The other three drive a single VM over the
forwarded SSH port.

QEMU user-mode networking does not let either pair see the other, so the
rendezvous has to go out to the public DHT and come back. That works
most of the time, which is why these two are the legs that redden
intermittently on master while the rest stay green on the same run and
the same image.

reusable-qemu-test.yaml already carries everything needed to avoid it: a
use_bridge_network input, a step that starts the libvirt default network
and allows virbr0 in the QEMU bridge helper, and a suite that reads
USE_BRIDGE_NETWORK and adds the second NIC. No caller ever set the
input, so none of it has run. master.yaml and pr-full.yaml now set it
for those two legs through a bridge key on the matrix entry.

A bridge that cannot be set up is now an error rather than a logged
warning. Before this the VM started on user-mode networking anyway and
the spec failed ten minutes later at its own rendezvous timeout, naming
discovery rather than the missing bridge.

Fixes kairos-io#5494

Signed-off-by: Ettore Di Giacinto <mudler@kairos.io>
@mudler
mudler merged commit e6dd68f into kairos-io:master Oct 20, 2026
24 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

CI: provider-qrcode-install and provider-decentralized-k8s regressed on master in ce57f7fc..4433f215

2 participants