Skip to content

BuildBackendHookCaller.subprocess_runner() context manager is not thread-safeΒ #226

Closed

Description

@gaborbernat

Description

BuildBackendHookCaller.subprocess_runner() context manager mutates self._subprocess_runner on the instance without any synchronization:

@contextmanager
def subprocess_runner(self, runner):
    prev = self._subprocess_runner
    self._subprocess_runner = runner
    try:
        yield
    finally:
        self._subprocess_runner = prev

If two threads share the same BuildBackendHookCaller instance and one of them uses subprocess_runner() to temporarily swap the runner, the other thread sees the swapped runner and may invoke hooks with the wrong runner. The restore in finally also races with concurrent reads of self._subprocess_runner in _call_hook.

Impact

This prevents consumers (e.g. pypa/build) from safely sharing a BuildBackendHookCaller (or a wrapping ProjectBuilder) across threads when subprocess_runner() is used.

Discovered during a thread-safety audit of pypa/build for pypa/build#377.

Activity

  1. takluyver commented on Apr 9, 2026

    @takluyver
    Member

    BuildBackendHookCaller should be pretty cheap to create. Is it reasonable to document that multi-threaded callers should create a separate instance per thread, rather than adding something like a thread local? Or even just document that you shouldn't use the convenience context manager along with threads? You can set the context manager subprocess runner when instantiating the object.

    FWIW, I generally don't assume that arbitrary Python objects are thread safe unless it's documented as such.

  2. hansu650 commented on May 18, 2026

    @hansu650

    I checked this locally and can reproduce the cross-thread runner leak with a small regression test: while one thread is inside subprocess_runner(scoped_runner), another thread using the same caller also sees the scoped runner.

    I also have a small ContextVar-based patch that keeps the temporary runner scoped to the current context while preserving the default runner on the shared caller.

    Before opening a PR, would you prefer a code fix for this behavior, or would you rather document that callers should use a separate BuildBackendHookCaller per thread?

  3. takluyver commented on May 19, 2026

    @takluyver
    Member

    I would rather document this for now.

    If the limitation is causing actual difficulties, then I would rather design an API that visibly doesn't rely on mutable state, instead of adding magic mechanisms around that state. That would look something like this:

    new_hook_caller = hook_caller.with_subprocess_runner(my_runner)  # Make a modified copy
    new_hook_caller.get_requires_for_build_wheel()

    But it doesn't look like build actually uses the with hook_caller.subprocess_runner(...) API that this issue is about, so I'm not sure there actually is a problem at all.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions