Repository navigation
fix: make the child subreaper opt-in instead of always on - #8
Merged
Merged
Conversation
Run() made the calling process a child subreaper and started a goroutine reaping with wait(-1) every 100ms. Both are process-wide: the reaper claims any child of the calling process, not only the orphans it is meant to collect, so it races the Go runtime for children started with os/exec anywhere else in the program. When it wins, an ordinary exec.Command(...).Run() that ran fine loses its exit status and returns "waitid: no child processes". The subreaper arrived in mudler#3 as extra hardening for containers. The part of that change which fixed the reported leak, the process-group kill and the graceful timeout, is untouched here; only the reaping is now behind WithSubreaper(true), off by default. A program that is the init of its own PID namespace can opt back in; a program that merely runs a subprocess no longer has os/exec broken underneath it. Signed-off-by: Ettore Di Giacinto <mudler@kairos.io>
Contributor
Author
|
@mudler this is a fork PR, so its |
mudler
approved these changes
Sep 29, 2026
mudler
pushed a commit
to kairos-io/kairos
that referenced
this pull request
Sep 30, 2026
…ming exec children (#5082) v0.1.1 made the test process a child subreaper on every process.Run() and started a goroutine polling Wait4(-1, WNOHANG) every 100ms for the life of the VM. Wait4(-1) claims any child of the caller, not only the orphans it is meant to collect, so every exec.Command(...).CombinedOutput() in the suite raced it. When the reaper won, os/exec found nothing left to wait for and returned "waitid: no child processes" for a command that had already run and whose output was captured in full. That is the intermittent test-standard-encryption red. v0.1.2 is mudler/go-processmanager#8: the subreaper and the wildcard reaping are behind WithSubreaper(true) and default off. peg constructs its qemu process without that option, so the reaping no longer happens. The process group kill and the graceful stop timeout, which are what the subreaper was added for, are untouched. Part of #5043 Signed-off-by: ci-robbot <ci-robbot@users.noreply.github.com> Co-authored-by: ci-robbot <ci-robbot@users.noreply.github.com> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Run()makes the calling process a child subreaper and starts a goroutinethat reaps with
syscall.Wait4(-1, ...)every 100ms for the life of the managedprocess. Both are process-wide, and
Wait4(-1)claims any child of thecalling process, not only the orphans it is meant to collect.
So the reaper races the Go runtime for every child the embedding program starts
with
os/exec. When the reaper wins, the child is already gone by the timeos/execwaits for it, and anexec.Command(...).Run()that ran perfectlyreturns:
with its output fully captured and its exit status lost. A library that runs a
subprocess should not break
os/execfor the rest of the program.This turns the reaping into an opt-in,
WithSubreaper(true), off by default.Why off by default
The subreaper arrived in #3 as extra container hardening. The part of that
change which fixed the leak it was filed for (LocalAI#7958) is the process-group
kill and the graceful timeout, and both are untouched here. Only the
Wait4(-1)reaping moves behind the flag.A program that really is the init of its own PID namespace still wants it and
can ask for it. A program that merely runs a subprocess, which is the common
case, gets working
os/execback.SetSubreaper()stays exported for anyonecalling it directly.
Callers who want the old behaviour:
process.New(..., process.WithSubreaper(true)).How it was found
kairos-io/kairose2e tests usepeg, which uses this library to launch qemu.Their encryption suite intermittently failed on a
kubectlinvocation that hadplainly succeeded, printing the complete certificate it had just fetched
alongside
waitid: no child processes.How it was tested
Reproduced first, standalone, no VM involved:
Same binary with
p.Run()skipped: 0/20000. The captured output is alwayscomplete; only the wait fails.
subreaper_test.gocommits that as two deterministic specs. They sidestep therace by widening the window past the reaper's 100ms poll:
Start(), sleep500ms, then
Wait(). A running reaper has certainly claimed the child by then.exec.Commandstill reports its child's exit statusWithSubreaper(true): it does not, which is the documented cost of opting inBoth mutation checked:
[Fail] Subreaper by default [It] leaves os/exec in the host program alone/waitid: no child processes[Fail] Subreaper when the caller asks for it [It] reaps orphans, and so also claims the host program's childrenSuite green with the fix in place:
15 Passed | 0 Failed.go build ./...andgo vet ./...are clean.gofmtreportsconfig.go,process_unix.goandwriter.go, but all three are already unformatted onmaster, so I left them rather than mixing a reformat into this diff.I could not run
go test -racelocally, as this machine has no C compiler; CIcovers it. The change adds no concurrency, it removes a goroutine in the default
path.
AI disclosure
Written by an AI agent. No human reviewed the code before this pull request was
opened. The reproduction, the mutation checks and the test runs above were all
actually executed.