Skip to content

feat: support for Go's native FIPS 140-3 module - #4522

Merged
jimmykarily merged 1 commit into
kairos-io:masterfrom
klaymen47:feat/native-fips-140-3
Sep 14, 2026
Merged

jimmykarily merged 1 commit into
kairos-io:masterfrom
klaymen47:feat/native-fips-140-3

Conversation

@klaymen47

Copy link
Copy Markdown
Contributor

This PR implements #4502.

Since Go's native FIPS module does not rely on CGO at all, I've also done some cleanup related to cross-compiling FIPS for arm64 and the C toolchain itself.

One important limitation - I've only added the metadata check for whether FIPS module is enabled or not. The runtime check is still to be done - I think it's best to implement it under a separate issue. This approach allows for this particular PR to focus mostly on build/CI-related matters.

Same goes for riscv64 support which is mentioned in #4419.

Please kindly review and provide your feedback :)

@klaymen47
klaymen47 requested a review from a team as a code owner September 7, 2026 13:52
@itxaka-agent

Copy link
Copy Markdown
Collaborator

Automated triage agent (kairos-triage-agent) running as @itxaka-agent.
This comment was generated by software. A human maintainer can override
any action taken here.

Taking this over for review. Pulling the branch, reading the diff against #4502 and the riscv64-scoping note in #4419, and will post a verdict shortly.

@klaymen47 klaymen47 changed the title Add support for Go's native FIPS 140-3 module feat: support for Go's native FIPS 140-3 module Sep 7, 2026

@itxaka-agent itxaka-agent left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated triage agent (kairos-triage-agent) running as @itxaka-agent.
This comment was generated by software. A human maintainer can override
any action taken here.

Verdict: changes-requested.

The FIPS migration itself is sound. Dropping the arm64 cross-compiler install and CC=aarch64-linux-gnu-gcc is safe since these builds already run with CGO_ENABLED=0, and the new verify-fips Makefile target does propagate a non-zero exit correctly and fails closed when the dist directory is empty or missing.

Two blocking findings are inline on the diff below: the deleted osv-scanner normalization step in release-scan.yml only covers half of the case it used to filter (a nodwarf5-suffixed version from the bundled edgevpn binary is untouched by this PR and would still trip the release gate), and the new --fips validation check in kairos-init/pkg/validation/validate.go can never run because --fips is registered as a local flag on the root command, not passed through to the validate subcommand. Two smaller riders: kcrypt/.goreleaser.yaml drops CGO_LDFLAGS="-ldl" from the non-FIPS build too, outside this PR's stated scope, and there's a stray extra blank line in the Makefile.

No new tests are owed here (rule 14): this is CI/build-config work verified by the new verify-fips target running in CI, not by Go unit tests. The one Go code change in validate.go has no practical unit-test seam without a larger refactor, so the review asks for the flag to be reachable rather than for new test coverage.

One residual uncertainty I could not close from this workspace: I cannot confirm offline that a Go 1.26.8 binary built with GOFIPS140=v1.0.0 actually emits the exact go version -m substrings the new verify-fips check greps for (GOFIPS140=v1.0.0 and a DefaultGODEBUG= line containing fips140=on). The check is fail-closed, so a mismatched pattern would turn the FIPS build matrix red rather than pass silently -- worth a first real CI run to confirm.

Comment thread .github/workflows/release-scan.yml Resolved
Comment thread kairos-init/pkg/validation/validate.go Resolved
Comment thread kcrypt/.goreleaser.yaml Resolved
Comment thread Makefile Resolved
@itxaka-agent

Copy link
Copy Markdown
Collaborator

Automated triage agent (kairos-triage-agent) running as @itxaka-agent.
This comment was generated by software. A human maintainer can override
any action taken here.

Audit summary -- round 0

Phase Role What happened
Intake manager Pulled the branch (review-repro/4522, head e22609b6) against upstream/master, cross-referenced the linked feature request (#4502) and the riscv64 tracking epic (#4419) referenced from this PR's own body.
Review reviewer Read the full diff (Makefile, three .goreleaser.yaml files, two CI workflows, release-scan.yml, validate.go) plus the surrounding source it touches (main.go's flag wiring, Dockerfile.test, steps_init.go, the bundled-binaries fetch script). Verdict: changes-requested.
Publish manager Verified all four inline comment line numbers against the diff hunks directly, then posted the review with 4 inline findings via the GitHub Reviews API.

Findings (all posted inline on the diff)

  1. .github/workflows/release-scan.yml:119 -- the deleted osv-scanner normalization step covered two version-string suffixes (-X:boringcrypto and -X:nodwarf5), but this PR only retires the boringcrypto one. The bundled edgevpn binary (fetched prebuilt, not built by this repo) can still carry a nodwarf5-style suffix, so a stdlib advisory that's actually already fixed could once again fail the release-gating job.
  2. kairos-init/pkg/validation/validate.go:162 -- the new FIPS validation check can never run. --fips is registered as a local flag on the root command in main.go, not wired into the validate subcommand's shared flags, so kairos-init validate --fips errors out and kairos-init validate (as called from Dockerfile.test) always sees Fips=false.
  3. kcrypt/.goreleaser.yaml:6 -- the change also drops CGO_LDFLAGS="-ldl" from the non-FIPS build, which is outside this PR's stated scope and unexplained in the commit message.
  4. Makefile:184 -- a stray extra blank line left after the new verify-fips target.

No new tests were requested (rule 14): this is CI/build-config work verified by the PR's own new verify-fips target running in CI, and the one Go code change has no practical unit-test seam without a larger refactor -- the review asked for the flag to be reachable instead.

The PR stays assigned to nobody and unmerged; this ticket moves to awaiting-author and will be re-reviewed automatically once the author pushes or responds.

Signed-off-by: klaymen47 <klaymen47@gmail.com>

@itxaka-agent itxaka-agent left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated triage agent (kairos-triage-agent) running as @itxaka-agent.
This comment was generated by software. A human maintainer can override
any action taken here.

Round 1 review. Three of the four round-0 findings are resolved: the unreachable --fips validation branch is gone (replaced with an accurate TODO), the double blank line in the Makefile is fixed, and the kcrypt/.goreleaser.yaml CGO_LDFLAGS="-ldl" removal turns out to be correctly scoped after all (my round-0 read of that hunk was wrong โ€” it stays on the default build). One finding is still open: the Normalize GOEXPERIMENT stdlib versions step in release-scan.yml is still deleted outright rather than narrowed, and it still guards a real case (the prebuilt edgevpn binary) that this PR doesn't touch.

Comment thread .github/workflows/release-scan.yml Resolved
@itxaka-agent

Copy link
Copy Markdown
Collaborator

Automated triage agent (kairos-triage-agent) running as @itxaka-agent.
This comment was generated by software. A human maintainer can override
any action taken here.

Audit trail โ€” round 1

Phase Round Role Result
reviewing 0 reviewer changes-requested (4 findings)
awaiting-author โ€” manager round-0 review + summary posted
reviewing 1 reviewer changes-requested (1 finding)

Round 1 re-reviewed the branch after two force-pushes (e22609b6 โ†’ 0d520fec โ†’ 9f46904c). Three of the four round-0 findings are resolved:

  • kairos-init/pkg/validation/validate.go โ€” the unreachable --fips branch was replaced with an accurate TODO explaining the flag-plumbing gap.
  • Makefile โ€” the extra blank line at the verify-fips target boundary is gone.
  • kcrypt/.goreleaser.yaml โ€” on closer look this one was a misread in round 0: CGO_LDFLAGS="-ldl" does still stay on the default build; only the two FIPS builds drop it alongside the GOEXPERIMENTโ†’GOFIPS140 swap. No author action was needed here.

One finding carries over unresolved, posted inline again this round:

  • .github/workflows/release-scan.yml:119 โ€” the Normalize GOEXPERIMENT stdlib versions step is still deleted outright. Half the premise holds (GOFIPS140 doesn't add a version suffix, unlike GOEXPERIMENT=boringcrypto), but the step also covered a second, unrelated case: the prebuilt edgevpn binary bundled from a different project, whose version string this PR doesn't touch. Removing the whole step reopens that gate for a stdlib CVE match on edgevpn, and it's a release-blocking check, not just a PR-time one.

Ticket stays in awaiting-author. Next slot re-polls per rule 12b โ€” if the author pushes again, round 2 runs.

@itxaka-agent

Copy link
Copy Markdown
Collaborator

Automated triage agent (kairos-triage-agent) running as @itxaka-agent.
This comment was generated by software. A human maintainer can override any action taken here.

@Itxaka @klaymen47 โ€” you're right, my round-1 finding was wrong. Verified locally with Go 1.25.4:

== plain ==
hi-plain: go1.25.4

== GOFIPS140=v1.0.0 ==
hi-fips: go1.25.4                              โ† no suffix
  build   -tags=fips140v1.0
  build   DefaultGODEBUG=fips140=on
  build   GOFIPS140=v1.0.0-c2097c7c

== GOEXPERIMENT=boringcrypto ==
hi-bc:   go1.25.4 X:boringcrypto               โ† this is what needed normalizing
  build   GOEXPERIMENT=boringcrypto

The Normalize GOEXPERIMENT stdlib versions step existed specifically because GOEXPERIMENT=boringcrypto appends X:boringcrypto to the go version output. GOFIPS140 records itself as a separate build setting instead and leaves the version string alone, so the normalization has nothing to do. Deleting the step is correct.

Retracting the round-1 finding and approving in the next comment. Apologies for the noise.

@itxaka-agent itxaka-agent left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated triage agent (kairos-triage-agent) running as @itxaka-agent.
This comment was generated by software. A human maintainer can override any action taken here.

Retracting the round-1 Normalize GOEXPERIMENT stdlib versions finding โ€” verified locally that GOFIPS140 does not append X:... to the Go version string, so the normalize step has nothing to normalize and the deletion is correct. See the comment I just posted for the empirical trace.

All prior findings are addressed. No blocking issues remain from my side; approving.

@Itxaka

Itxaka commented Sep 7, 2026

Copy link
Copy Markdown
Member

Automated triage agent (kairos-triage-agent) running as @itxaka-agent.
This comment was generated by software. A human maintainer can override any action taken here.

@Itxaka @klaymen47 โ€” you're right, my round-1 finding was wrong. Verified locally with Go 1.25.4:

== plain ==
hi-plain: go1.25.4

== GOFIPS140=v1.0.0 ==
hi-fips: go1.25.4                              โ† no suffix
  build   -tags=fips140v1.0
  build   DefaultGODEBUG=fips140=on
  build   GOFIPS140=v1.0.0-c2097c7c

== GOEXPERIMENT=boringcrypto ==
hi-bc:   go1.25.4 X:boringcrypto               โ† this is what needed normalizing
  build   GOEXPERIMENT=boringcrypto

The Normalize GOEXPERIMENT stdlib versions step existed specifically because GOEXPERIMENT=boringcrypto appends X:boringcrypto to the go version output. GOFIPS140 records itself as a separate build setting instead and leaves the version string alone, so the normalization has nothing to do. Deleting the step is correct.

Retracting the round-1 finding and approving in the next comment. Apologies for the noise.

good bot

itxaka-agent added a commit to Itxaka/itxaka-agent that referenced this pull request Sep 7, 2026
The reviewer is read-only on purpose. It cannot compile anything,
so any finding whose truth depends on how a compiler or linker
actually behaves at build time was previously reasoned-out from
source-reading and shipped as a live blocker. That failure mode
just cost us a wrong review on kairos-io/kairos#4522: the reviewer
said the `Normalize GOEXPERIMENT stdlib versions` step in
release-scan.yml would keep needing to run under GOFIPS140, the
author built the bundle locally and posted 7 binaries with no
version suffix, and Itxaka asked us to verify locally. The claim
was already contradicted by empirical output the reviewer had
read; the review shipped anyway because we had no path from
"reviewer suspects X" to "toolchain observably does X" other
than the reviewer's own reasoning.

Add a fifth role, `kairos-triage-builder`. It gets Bash, reads
one `build_claim` per invocation, writes the smallest possible
program that either confirms or contradicts the claim, and
reports what its output actually says. The reviewer flags a
finding with `needs_build_verification: true` and a testable
`build_claim` sentence; the manager dispatches the builder
before publication and routes the verdict:

- confirmed -> the finding survives and the builder's one-line
  goes into the published comment as evidence
- contradicted -> the finding is dropped, the audit trail records
  the retraction, the PR author never sees the wrong claim
- inconclusive -> the finding is dropped for this round with a
  note and can be re-raised next round with more context

Also promote `changes-requested` to `approve` when all the
round's findings get dropped this way, so a review whose sole
blocker was a toolchain misread ends up approved instead of stuck
at awaiting-author.

Ships:
- .claude/agents/kairos-triage-builder.md
- rule 9b in RULES.md
- `needs_build_verification` + `build_claim` in the reviewer
  verdict JSON
- builder dispatch step in the manager's reviewing phase
- roles.builder_wallclock_seconds config knob (default 900)
- entry in docs/agent-roles.md

Signed-off-by: itxaka-agent <itxaka-agent@users.noreply.github.com>
@ci-robbot

Copy link
Copy Markdown
Collaborator

Triage: approved with the whole matrix green except two cells, and neither is your diff. Both died in the harness before a single spec ran.

  • encryption-remote-ek-only: k3d cluster create failed with docker failed to start container and rolled itself back, so .github/encryption-tests.sh exited at line 61 (job). That is the one setup step in that script still not wrapped in its own retry helper; I am sending a patch for it.
  • provider-upgrade-latest-k8s-with-kubernetes: HTTP 500 from api.github.com downloading the v4.3.0-k3sv1.36.4+k3s1 release asset (job).

Worth noting for the reviewers that build-*/build (amd64, fips) and (arm64, fips) are green here, which is the part this PR actually changes. Re-run the two cells and this looks ready for QA.

@ci-robbot

Copy link
Copy Markdown
Collaborator

Picking this up for QA.

@ci-robbot

Copy link
Copy Markdown
Collaborator

QA Result: PASS

Check Result
Environment PR head 9f46904c merged onto upstream/master 58b5d404 (clean, merge 0e4fa757). Go 1.26.8, go.mod go 1.26.8, CGO_ENABLED=0. Host has no aarch64-linux-gnu-gcc and no crossbuild-essential-arm64, matching a CI runner after this PR drops the apt step. osv-scanner v2.5.0. QEMU/KVM, UEFI (OVMF), 4G, 4 vCPU.
Reproduction All three steps from #4502, then the whole thing end to end as a FIPS user consumes it. Step 1: built all four binaries (kairos, kcrypt-challenger, kairos-installer, provider-kairos) for VARIANT=fips on amd64 and arm64. Step 2: make verify-fips positive, plus three negative cases. Step 3: ran the exact release-scan osv-scanner invocation with the normalization step gone. Then make binaries (the release-scan path), kairos-init validate on FIPS and non-FIPS images, FIPS OS image -> ISO -> boot live ISO -> real TLS image pull in the guest -> manual-install to /dev/vda -> reboot from disk -> re-verify on the installed node.
Expected FIPS binaries are built against the certified Go native FIPS 140-3 module (GOFIPS140=v1.0.0) instead of GOEXPERIMENT=boringcrypto; arm64 FIPS cross-builds with no C toolchain; the build-time check actually fails when FIPS is absent; the release-scan normalization step is dead weight; and a FIPS image still installs, boots and does crypto.
Actual Matches on every point. GOFIPS140=v1.0.0-c2097c7c and DefaultGODEBUG=fips140=on are baked into every FIPS binary, verify-fips passes on real FIPS binaries and fails closed on all three negative cases, arm64 FIPS cross-builds with no cross-compiler present, the scan is clean with 1235 packages extracted and zero findings, and the installed FIPS node boots to running with 0 failed units and completes a TLS image pull.
Proofs gist: verify-fips positive/negative runs, binary metadata and sha256 chain, make binaries gate output, release-scan results, and the full serial console logs for the live-ISO boot, the install, and the installed-node boot. Screenshots inline below.

Summary: the migration works and the new gate is genuinely load-bearing. Verified on a booted and installed FIPS node, not only in the build.

Step 1: the GOFIPS140 swap

Both arches, all four binaries, no C toolchain on the host:

$ which aarch64-linux-gnu-gcc
(absent)

$ make kairos kcrypt-challenger kairos-installer provider-kairos ARCH=arm64 VARIANT=fips
GOOS=linux GOARCH=arm64 CGO_ENABLED=0 GOFIPS140=v1.0.0 go build ...     <- no CC=, no apt step
real  0m39.777s

$ file dist/linux-arm64-fips/kairos
ELF 64-bit LSB executable, ARM aarch64, statically linked, stripped

This is the claim behind removing Install cross-compiler (FIPS-arm64) from both build workflows, and it holds.

Step 2: verify-fips is real, and it fails closed

$ make verify-fips ARCH=amd64 VARIANT=fips
verify-fips: dist/linux-amd64-fips/kairos OK
verify-fips: dist/linux-amd64-fips/kairos-installer OK
verify-fips: dist/linux-amd64-fips/kcrypt-challenger OK
verify-fips: dist/linux-amd64-fips/provider-kairos OK          <- exit 0

$ make verify-fips ARCH=amd64 VARIANT=fips DIST_ARCH_DIR=dist/linux-amd64     # default binaries
verify-fips: dist/linux-amd64/kairos not built with GOFIPS140=v1.0.0          <- exit 2

$ make verify-fips ARCH=amd64 VARIANT=fips DIST_ARCH_DIR=/tmp/emptydir
verify-fips: no binaries under /tmp/emptydir -- refusing to report success    <- exit 2

$ make verify-fips ARCH=amd64 VARIANT=default
verify-fips: VARIANT=default, nothing to verify                               <- exit 0

The empty-directory guard matters: without it the target would report success on a build that produced nothing. And it is wired into the path CI actually runs, inside the ifneq ($(ARCH),riscv64) guard, so riscv64 correctly skips both the FIPS build and the check:

$ make -n binaries ARCH=riscv64 | grep -E "verify-fips|VARIANT=fips"
(nothing)
$ make -n binaries ARCH=amd64  | grep -E "verify-fips|VARIANT=fips"
make kairos kcrypt-challenger kairos-installer provider-kairos ARCH=amd64 VARIANT=fips
make verify-fips ARCH=amd64 VARIANT=fips

Step 3: the release-scan normalization really is dead weight

Neither variant emits an -X: suffix any more, which was the whole reason the jq step existed:

kairos-init/pkg/bundled/binaries/kairos               go1.26.8   (no-fips)
kairos-init/pkg/bundled/binaries/fips/kairos          go1.26.8   GOFIPS140=v1.0.0
kairos-init/pkg/bundled/binaries/edgevpn              go1.26.7   (no-fips)

I ran the gate's own osv-scanner command on that tree with the step removed:

extracted packages (false-green guard): 1235      <- guard satisfied
stdlib versions seen: 1.26.7, 1.26.8              <- plain, no -X: suffix
findings: 0

No other -X: producer sneaked in, including the externally fetched edgevpn, so removing the block does not create a false red.

End to end: a FIPS node, built and booted

Chain of custody from the Makefile to the running system. The binary in the booted node is byte-for-byte the one verify-fips validated:

fdcd58a4797ca8a48799543e40154a3c36a50e3faf6756297114646fc3fccf11  dist/linux-amd64-fips/kairos
fdcd58a4797ca8a48799543e40154a3c36a50e3faf6756297114646fc3fccf11  (extracted from the FIPS OS image)
fdcd58a4797ca8a48799543e40154a3c36a50e3faf6756297114646fc3fccf11  /usr/bin/kairos on the booted node
2a21388473fb14c2d0e6c56a03326a08e66baa97707d982db249eb8874bc280a  dist/linux-amd64/kairos   <- default, different

Live ISO, logged in over the VGA console:

live ISO FIPS evidence

Then manual-install onto /dev/vda (INSTALL_RC=0), CD-ROM detached, booted from disk:

installed system FIPS evidence

$ cat /proc/cmdline
... root=LABEL=COS_ACTIVE cos-img/filename=/cOS/active.img ...   <- installed, not live media
$ sudo kairos-agent state | grep boot
boot: active_boot
$ systemctl is-system-running ; systemctl --failed --no-legend | wc -l
running
0

FIPS crypto exercised for real on that installed node, not just inspected. A registry pull is a TLS 1.3 handshake plus X.509 verification plus SHA-256 digest checking, all through the certified module:

$ sudo kairos-agent pull-image alpine:3.19 /tmp/p
INF Image alpine:3.19 downloaded and extracted to /tmp/p correctly
$ cat /tmp/p/etc/alpine-release
3.19.9

And the non-FIPS side is unchanged: KAIROS_FIPS="false", and the installed binary carries no GOFIPS140 and no fips140=on default.

Three things worth knowing, none of them blocking

1. The metadata value is not literally v1.0.0. go version -m reports GOFIPS140=v1.0.0-c2097c7c. Every check here is a grep -F substring match, so all of them match correctly. Flagging it only so nobody later "tightens" one of these into an anchored or exact comparison and breaks the gate.

2. A future module bump touches 18 hardcoded strings. The Makefile has the nice GOFIPS140_VERSION := v1.0.0 variable, but the three goreleaser files hardcode v1.0.0 in 18 places (env blocks and hook greps) rather than sharing it. Going to v1.0.1 means editing all of them, and missing a hook grep would leave a check asserting the old version. Not a defect in this PR, and arguably not worth solving until there is a second certified version, but worth a note.

3. fips140=only panics, exactly as the issue predicted. Out of scope per #4502, and confirmed pre-existing rather than caused by this diff. gofrs/uuid.NewV5 uses SHA-1 at package init in agent/pkg/constants/constants.go:226:

panic: crypto/sha1: use of SHA-1 is not allowed in FIPS 140-only mode
	github.com/kairos-io/kairos/v4/agent/pkg/constants.init()

The default (non-FIPS) binary panics identically under the same GODEBUG, so this is not introduced here. It does mean fips140=only is currently unreachable for Kairos, which supports the decision to scope it out.

Unrelated observations

kairos-init validate fails on both the FIPS and the non-FIPS image with the identical [INITRD] did not find kairos-agent in the initrd. This diff cannot cause it (the change to validate.go is purely additive to the release-keys list), and it reproduces on the non-FIPS image too, so it is pre-existing and out of scope here. Mentioning it because this PR touches that file. Notably there were no [RELEASE] errors on either image, so adding KAIROS_FIPS to the required keys is safe: steps_init.go:189 already writes it unconditionally as %t, giving "false" on non-FIPS builds rather than an empty value.

The two red checks are both infra, upstream of anything Kairos does:

  • provider-upgrade-latest-k8s-with-kubernetes: Download asset '...iso' failed: GitHub server error ... (HTTP 500), on releases/assets/550036180.
  • encryption-remote-ek-only: k3d cluster creation failed on docker bridge networking, bridge port not forwarding after 200ms, rolled back.

Neither reaches a Kairos boot. The other 67 checks are green.

mudler pushed a commit that referenced this pull request Sep 8, 2026
Every other setup step in this script goes through the retry helper, but
the cluster create does not. It is the step that fails: on #4522 docker
refused to start the k3s server container, and the encryption cell went
red before a single spec ran.

k3d rolls its own creation back on failure, so the retry starts from a
clean slate rather than from a half-built cluster.

Signed-off-by: ci-robbot <ci-robbot@users.noreply.github.com>
Co-authored-by: ci-robbot <ci-robbot@users.noreply.github.com>
@jimmykarily
jimmykarily merged commit 8d1f45f into kairos-io:master Sep 14, 2026
134 of 136 checks passed

This branch was previously deployed

1 inactive deployment
fork-pr-builds โ€” 9f46904c Deployed Sep 7, 2026 by klaymen47 via authorize #299
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/security enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants