Skip to content

perf(venv): skip the distutils shim on Python 3.10+ - #1358

Merged
jbedard merged 3 commits into
aspect-build:mainfrom
konsti-openai:konsti/remove-virtualenv-distutils-shim
Aug 13, 2026
Merged

jbedard merged 3 commits into
aspect-build:mainfrom
konsti-openai:konsti/remove-virtualenv-distutils-shim

Conversation

@konsti-openai

@konsti-openai konsti-openai commented Jul 20, 2026 •

Copy link
Copy Markdown
Contributor

This is a port of pypa/virtualenv#3184 and astral-sh/uv#20222, which removed the distutils hack recently. Removing this brings venv creation in line with virtualenv and uv again and improves Python startup perf.

This is the conservative version of this change. The workaround is only required in an obscure setup where you have the bogus ancient configuration file in the current directory when performing operations. If we can safely assume that bazel users don't do that, we can remove the two files entirely.


Changes are visible to end-users: yes

  • Searched for relevant documentation and updated as needed: yes
  • Breaking change (forces users to change their own code or config): yes
  • Suggested release notes appear below: yes

Python 3.9 venvs keep the shim.

Python 3.10+ targets:

  • _virtualenv.py / _virtualenv.pth are no longer created, and the venv bin/ PATH setup moved into the main <name>.pth.
  • Code importing _virtualenv directly (e.g. as a path anchor) must use another anchor such as sys.prefix.
  • On 3.10+, a direct legacy setup.py install into a built venv with underscore-form [install] keys (install_purelib, …) is no longer stripped and can install outside the venv. Installing into a built venv at runtime is not a supported rules_py workflow.

Test plan

  • Covered by existing test cases

@konsti-openai
konsti-openai force-pushed the konsti/remove-virtualenv-distutils-shim branch from e57b68d to 6d2673b Compare July 20, 2026 22:21
@jbedard
jbedard requested review from jbedard and xangcastle August 6, 2026 23:59
@jbedard

jbedard commented Aug 7, 2026

Copy link
Copy Markdown
Collaborator

@tamird review

@tamird

tamird commented Aug 7, 2026

Copy link
Copy Markdown
Contributor

@tamird review

what am I reviewing?

@jbedard

jbedard commented Aug 7, 2026

Copy link
Copy Markdown
Collaborator

@tamird review this PR

@tamird

tamird commented Aug 7, 2026

Copy link
Copy Markdown
Contributor

But why? Konsti works at OAI and is a core contributor to uv.

@jbedard

jbedard commented Aug 7, 2026

Copy link
Copy Markdown
Collaborator

@tamird because you provide valuable feedback, and I'm uncertain if someone being an OAI employee means they know how a bazel change will effect your full build/CI?

@tamird

tamird commented Aug 7, 2026

Copy link
Copy Markdown
Contributor

One compatibility case this doesn’t cover: setuptools/distutils checks its venv ignore list before normalizing option names, so underscore-form [install] settings such as install_purelib and install_scripts bypass the built-in guard. The existing _virtualenv.py hook is what strips them. Removing that hook on Python β‰₯3.10 therefore changes direct legacy setup.py install behavior even though ordinary pip paths stay green. Could we preserve that protection, cover the underscore-form settings, or explicitly document dropping that compatibility?

Separately, this changes the observable virtualenv layout: _virtualenv.py and _virtualenv.pth disappear, and the main .pth changes. We have downstream Python 3.12 image-layout goldens outside the ordinary CI path, so those will need a coordinated update.

@jbedard

jbedard commented Aug 10, 2026

Copy link
Copy Markdown
Collaborator

Please checkout the failing test.

The workaround is only required in an obscure setup where you have the bogus ancient configuration file in the current directory when performing operations. If we can safely assume that bazel users don't do that, we can remove the two files entirely.

Since we are still in v2 alpha maybe we can remove those entirely, how does it effect the tests? Do we know how common this is though? @xangcastle

@acozzette

Copy link
Copy Markdown
Collaborator

I believe the e2e tests can be fixed by running bazel run //:snapshots in e2e/cases. The examples/py_venv failures I'm not sure about, though.

Stop materializing and importing the virtualenv distutils shim for Python 3.10+, where pip and setuptools already ignore the affected install configuration. Keep the shim for Python 3.9 and move the rules_py-specific PATH setup into the existing venv .pth so console scripts remain discoverable.

Match the upstream changes:
pypa/virtualenv#3184
astral-sh/uv#20222
@jbedard
jbedard force-pushed the konsti/remove-virtualenv-distutils-shim branch from 6d2673b to f513b1f Compare August 13, 2026 02:55
@coderabbitai

coderabbitai Bot commented Aug 13, 2026 •

Copy link
Copy Markdown

Review Change Stack

No actionable comments were generated in the recent review. πŸŽ‰

ℹ️ Recent review info
βš™οΈ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 9e69eb0b-6596-4bb2-abab-d9edb65a619d

πŸ“₯ Commits

Reviewing files that changed from the base of the PR and between 043c235 and 12882b7.

β›” Files ignored due to path filters (1)
  • py/private/py_venv/tests/snapshots/venv.tree.snap is excluded by !**/*.snap
πŸ“’ Files selected for processing (25)
  • e2e/cases/cross-repo-610/test.py
  • e2e/cases/distutils-shim-39/BUILD.bazel
  • e2e/cases/distutils-shim-39/test.py
  • e2e/cases/interpreter-features-tkinter/snapshots/amd64_layers_listing.yaml
  • e2e/cases/oci/py_image_layer/snapshots/my_app_layers_fp_listing.yaml
  • e2e/cases/oci/py_image_layer/snapshots/my_app_layers_listing.yaml
  • e2e/cases/oci/py_image_layer/snapshots/my_app_layers_multi_listing.yaml
  • e2e/cases/oci/py_image_layer/snapshots/my_app_owner_listing.yaml
  • e2e/cases/oci/py_image_layer/snapshots/nested_external_layers_listing.yaml
  • e2e/cases/oci/py_image_layer/snapshots/scalar_data_layers_listing.yaml
  • e2e/cases/oci/py_venv_image_layer/snapshots/my_app_amd64_layers_listing.yaml
  • e2e/cases/oci/py_venv_image_layer/snapshots/my_app_arm64_layers_listing.yaml
  • e2e/cases/snapshots/firebase_admin.test_tool.venv.pth
  • e2e/cases/snapshots/multi_addsitedir.venv.pth
  • e2e/cases/snapshots/pth_install_tree_fallback.venv.pth
  • e2e/cases/snapshots/pth_install_tree_fallback_namespace.venv.pth
  • e2e/cases/snapshots/pth_namespace_547.test.venv.pth
  • e2e/cases/snapshots/wheel_root_pth.venv.pth
  • e2e/cases/uv-deps-650/crossbuild/snapshots/app_amd64_layers_listing.yaml
  • e2e/cases/uv-deps-650/crossbuild/snapshots/app_arm64_layers_listing.yaml
  • examples/py_venv/say.py
  • py/private/py_venv/templates/_virtualenv.py
  • py/private/py_venv/venv.bzl
  • py/tests/py-internal-venv/test.py
  • py/tests/venv-site-config/site_config_test.py
πŸ’€ Files with no reviewable changes (1)
  • py/private/py_venv/templates/_virtualenv.py

πŸ“ Walkthrough

Walkthrough

The venv generator now configures PATH through .pth startup code and emits the distutils shim only for Python versions below 3.10. Tests, examples, and image snapshots reflect the new behavior.

Changes

Virtual environment startup behavior

Layer / File(s) Summary
Venv generation and startup path setup
py/private/py_venv/venv.bzl, e2e/cases/snapshots/*.venv.pth
.pth startup code prepends the venv binary directory to PATH. _virtualenv.py and _virtualenv.pth are generated only for Python versions below 3.10.
Legacy shim coverage and updated assertions
e2e/cases/distutils-shim-39/*, py/tests/py-internal-venv/test.py, py/tests/venv-site-config/site_config_test.py, e2e/cases/cross-repo-610/test.py
Python 3.9 tests cover shim loading and distutils path protection. Python 3.10+ tests no longer require _virtualenv at startup.
Example and generated snapshot updates
examples/py_venv/say.py, e2e/cases/*/snapshots/*
The example derives paths from sys.prefix. Snapshots record larger .pth files and removed shim files.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Mergeability Score: πŸ”΅ Low Β· up to 12882

On Python 3.10+, direct legacy setup.py installs may no longer be prevented from placing files outside the venv when underscore-form install settings are used. This is a documented, unsupported workflow, but it remains a bounded compatibility risk that should have explicit owner acceptance or targeted test coverage.

Suggested reviewers: xangcastle, jbedard

πŸš₯ Pre-merge checks | βœ… 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 50.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
βœ… Passed checks (4 passed)
Check name Status Explanation
Title check βœ… Passed The title clearly summarizes the main change: skipping the distutils shim for Python 3.10 and newer.
Description check βœ… Passed The description directly explains the shim change, compatibility impact, affected Python versions, and test plan.
Linked Issues check βœ… Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check βœ… Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches πŸ’‘ 1
πŸ› οΈ Fix failing CI checks πŸ’‘
  • Create stacked PR
  • Commit on current branch
πŸ§ͺ Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands.

Fix references the shim-gating commit missed: tests asserting the shim
is loaded, say.py using it as a path anchor, the venv tree snapshot
(regenerated at the 3.13 default), venv .pth snapshots gaining the PATH
line, and the oci layer listings still containing shim entries.
@jbedard
jbedard force-pushed the konsti/remove-virtualenv-distutils-shim branch from f513b1f to 6a0db53 Compare August 13, 2026 03:58
Assert a 3.9 venv loads the shim at startup, materializes it as a real
file, and that its distutils patch clamps [install] prefix and strips
underscore-form keys the built-in venv guard misses.
@jbedard
jbedard merged commit 0648d0c into aspect-build:main Aug 13, 2026
94 of 98 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants