Skip to content

fix: make the child subreaper opt-in instead of always on - #8

Merged
mudler merged 1 commit into
mudler:masterfrom
ci-forks:bot/subreaper-opt-in
Sep 29, 2026
Merged

mudler merged 1 commit into
mudler:masterfrom
ci-forks:bot/subreaper-opt-in

Conversation

@ci-robbot

Copy link
Copy Markdown
Contributor

Summary

Run() makes the calling process a child subreaper and starts a goroutine
that reaps with syscall.Wait4(-1, ...) every 100ms for the life of the managed
process. Both are process-wide, and Wait4(-1) claims any child of the
calling process, not only the orphans it is meant to collect.

So the reaper races the Go runtime for every child the embedding program starts
with os/exec. When the reaper wins, the child is already gone by the time
os/exec waits for it, and an exec.Command(...).Run() that ran perfectly
returns:

waitid: no child processes

with its output fully captured and its exit status lost. A library that runs a
subprocess should not break os/exec for the rest of the program.

This turns the reaping into an opt-in, WithSubreaper(true), off by default.

Why off by default

The subreaper arrived in #3 as extra container hardening. The part of that
change which fixed the leak it was filed for (LocalAI#7958) is the process-group
kill and the graceful timeout, and both are untouched here. Only the
Wait4(-1) reaping moves behind the flag.

A program that really is the init of its own PID namespace still wants it and
can ask for it. A program that merely runs a subprocess, which is the common
case, gets working os/exec back. SetSubreaper() stays exported for anyone
calling it directly.

Callers who want the old behaviour: process.New(..., process.WithSubreaper(true)).

How it was found

kairos-io/kairos e2e tests use peg, which uses this library to launch qemu.
Their encryption suite intermittently failed on a kubectl invocation that had
plainly succeeded, printing the complete certificate it had just fetched
alongside waitid: no child processes.

How it was tested

Reproduced first, standalone, no VM involved:

p := process.New(process.WithName("/bin/sleep"), process.WithArgs("60"),
	process.WithStateDir(dir))
p.Run()

for i := 0; i < 20000; i++ {
	out, err := exec.Command("/bin/echo", "hello").CombinedOutput()
	if err != nil {
		fmt.Printf("output=%q ERR=%v\n", string(out), err)
	}
}
iter  2964: output="hello\n"  ERR=waitid: no child processes
iter  7120: output="hello\n"  ERR=waitid: no child processes
iter 17586: output="hello\n"  ERR=waitid: no child processes

3/20000 failed

Same binary with p.Run() skipped: 0/20000. The captured output is always
complete; only the wait fails.

subreaper_test.go commits that as two deterministic specs. They sidestep the
race by widening the window past the reaper's 100ms poll: Start(), sleep
500ms, then Wait(). A running reaper has certainly claimed the child by then.

  • default: exec.Command still reports its child's exit status
  • WithSubreaper(true): it does not, which is the documented cost of opting in

Both mutation checked:

mutation result
both gates forced on (the behaviour before this PR) [Fail] Subreaper by default [It] leaves os/exec in the host program alone / waitid: no child processes
both gates forced off (option wired to nothing) [Fail] Subreaper when the caller asks for it [It] reaps orphans, and so also claims the host program's children

Suite green with the fix in place: 15 Passed | 0 Failed.

go build ./... and go vet ./... are clean. gofmt reports config.go,
process_unix.go and writer.go, but all three are already unformatted on
master, so I left them rather than mixing a reformat into this diff.

I could not run go test -race locally, as this machine has no C compiler; CI
covers it. The change adds no concurrency, it removes a goroutine in the default
path.

AI disclosure

Written by an AI agent. No human reviewed the code before this pull request was
opened. The reproduction, the mutation checks and the test runs above were all
actually executed.

Run() made the calling process a child subreaper and started a goroutine
reaping with wait(-1) every 100ms. Both are process-wide: the reaper
claims any child of the calling process, not only the orphans it is
meant to collect, so it races the Go runtime for children started with
os/exec anywhere else in the program. When it wins, an ordinary
exec.Command(...).Run() that ran fine loses its exit status and returns
"waitid: no child processes".

The subreaper arrived in mudler#3 as extra hardening for containers. The part
of that change which fixed the reported leak, the process-group kill and
the graceful timeout, is untouched here; only the reaping is now behind
WithSubreaper(true), off by default. A program that is the init of its
own PID namespace can opt back in; a program that merely runs a
subprocess no longer has os/exec broken underneath it.

Signed-off-by: Ettore Di Giacinto <mudler@kairos.io>
@ci-robbot

Copy link
Copy Markdown
Contributor Author

@mudler this is a fork PR, so its Test run is parked at action_required and has never executed. Could you approve the workflow run? It is the last thing blocking the kairos e2e fix in kairos-io/kairos#5043, which needs a tag carrying this change.

@mudler
mudler merged commit e825bd0 into mudler:master Sep 29, 2026
mudler pushed a commit to kairos-io/kairos that referenced this pull request Sep 30, 2026
…ming exec children (#5082)

v0.1.1 made the test process a child subreaper on every process.Run() and
started a goroutine polling Wait4(-1, WNOHANG) every 100ms for the life of the
VM. Wait4(-1) claims any child of the caller, not only the orphans it is meant
to collect, so every exec.Command(...).CombinedOutput() in the suite raced it.
When the reaper won, os/exec found nothing left to wait for and returned
"waitid: no child processes" for a command that had already run and whose
output was captured in full. That is the intermittent
test-standard-encryption red.

v0.1.2 is mudler/go-processmanager#8: the subreaper and the wildcard reaping
are behind WithSubreaper(true) and default off. peg constructs its qemu
process without that option, so the reaping no longer happens. The process
group kill and the graceful stop timeout, which are what the subreaper was
added for, are untouched.

Part of #5043

Signed-off-by: ci-robbot <ci-robbot@users.noreply.github.com>
Co-authored-by: ci-robbot <ci-robbot@users.noreply.github.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants