Skip to content

perf: memoize check_dependency - #1122

Merged
gaborbernat merged 4 commits into
pypa:mainfrom
henryiii:perf/dependency-check-and-sdist-scan
Jul 7, 2026
Merged

gaborbernat merged 4 commits into
pypa:mainfrom
henryiii:perf/dependency-check-and-sdist-scan

Conversation

@henryiii

@henryiii henryiii commented Jul 1, 2026 •

Copy link
Copy Markdown
Contributor

Okay, I think this speedup is pretty clean, and can be seen for large projects:

Dependency tree main branch speedup
flit_core, setuptools, hatchling, pytest 0.02–1.3 ms ~same 1.0–1.1x
twine 11.2 ms 9.6 ms 1.16x
jupyter-server 45 ms 26 ms 1.7x
jupyterlab 214 ms 50 ms 4.3x

I removed the second change, as it wasn't measurably faster and was a costly change in code complexity.

🤖 Performance report 🤖

Measured both changes against main. Bottom line: the memoization is real but only matters in unusual cases; the sdist streaming change is a measured no-op.

check_dependency memoization

Dependency tree main branch speedup
flit_core, setuptools, hatchling, pytest 0.02–1.3 ms ~same 1.0–1.1x
twine 11.2 ms 9.6 ms 1.16x
jupyter-server 45 ms 26 ms 1.7x
jupyterlab 214 ms 50 ms 4.3x

Results were identical (same unmet-dependency sets) in every case. The 4.3x looks impressive, but context matters: this code path only runs with --no-isolation (isolated builds skip it, __main__.py:242), plus a tiny fixed build[uv] check in env.py:308. The trees actually walked are the build backend's requirements — hatchling, setuptools, flit_core — which are small and shallow. There the saving is ~0.1 ms per build. You'd need a backend (or a third-party caller of the public build.check_dependency) depending on something jupyterlab-shaped to save the ~165 ms, once, in builds that take seconds anyway.

Sdist streaming validation

Archive main (getmembers) branch (streaming) memory
1k files 8.2 ms 8.3 ms 1.00x
20k files 162 ms 164 ms 1.00x
100k files 818 ms 835 ms 1.00x

Zero benefit in both time and peak memory, and there's a structural reason: tarfile.open in the default r:* mode caches every TarInfo on tar.members as you iterate, so "streaming" iteration allocates exactly what getmembers() does. Only true stream mode (r|*) would change that — and even the absurd 100k-member case peaks at just 46 MB.

My take

  • Sdist change: drop it. It rewrites a clear 3-line validation into a partition/tracking loop (plus the new tests it needed) for a benefit that measures at 0.98–1.00x. The comment's claim about "materialising the full member list" is factually wrong for this open mode.
  • Memoization: borderline, lean drop. It triples the function count and carries a subtle invariant (only memoize satisfied subtrees, or unmet deps stop being reported per-chain) that's easy to break later. For build's own call sites the saving is microseconds. The only argument for keeping it is that check_dependency is public API and a --no-isolation user with an exotic backend gets protection from pathological diamond graphs — but 214 ms is the worst case I could construct with a real package, and that's hardly a problem worth guarding against.

If you want the branch to earn its keep, the memoization alone (with the marker-helper refactor folded back in) is the defensible half; I wouldn't ship the sdist part.

🤖 AI text below 🤖

Part of the code review in #1116.

Two behaviour-preserving performance improvements found in review.

check_dependency memoization (src/build/_util.py). The recursive dependency check only had a cycle guard, so a diamond dependency graph (A→B,C; B→D; C→D) re-walked each shared subtree once per path — exponential blowup on dense graphs, visible with --no-isolation dependency checks. A seen set of requirement strings already verified as satisfied is now threaded through an internal _check_dependency helper (public signature and yielded tuple format unchanged). Correctness subtlety: marker evaluation depends on parent_extras, but past the marker gate the outcome depends only on the requirement itself, so the memo lives past that gate and is keyed by the normalised requirement string. Only fully satisfied subtrees are recorded, so unmet requirements are still reported once per dependency chain. Regression test added (test_check_dependency_diamond_visits_each_once) asserting each package is looked up at most once; it fails before the fix (D looked up twice).

_validate_sdist_archive streaming (src/build/__main__.py). Replaced tar.getmembers() (which materialized a TarInfo for the whole archive, then iterated it twice) with a single streaming pass over the TarFile, accumulating top-level name segments and which of them carry a <top>/PKG-INFO regular file. Memory/CPU now proportional to one pass; all error messages and behaviour (including the empty-top-level discard() handling) are unchanged. Added test_validate_sdist_archive_ignores_empty_top_level covering the empty-segment edge.

All existing check_dependency and sdist-validation tests pass unchanged. pytest, mypy, and prek are green.

henryiii added 4 commits July 7, 2026 15:10
check_dependency recursively verified a requirement and its transitive
deps with only a cycle guard, so a diamond dependency graph re-walked
each shared subtree once per path (exponential on dense graphs, visible
with --no-isolation dependency checks). Thread a ``seen`` memo of
requirement strings already verified as satisfied through an internal
helper. Marker evaluation depends on parent_extras, but past the marker
gate the outcome depends only on the requirement itself, so memoising
past that gate by the normalised string is correct; only fully satisfied
subtrees are recorded so unmet requirements are still reported per chain.

_validate_sdist_archive called tar.getmembers() (materialising every
TarInfo) and iterated it twice. Stream the archive once, accumulating
top-level segments and which of them carry a <top>/PKG-INFO regular file,
for memory/CPU proportional to a single pass on large sdists. Behaviour
and error messages are unchanged.

Assisted-by: ClaudeCode:claude-opus-4.8
Assisted-by: ClaudeCode:claude-opus-4.8
CI enforces 100% coverage including test files. Exhaust the
check_dependency generator for an incompatible version and an unmet
transitive dependency (existing tests stop at the first yield via
``next``), add an empty-name tar member to exercise the streaming
skip, and drop a dead not-found branch in the diamond test mock.
Shorten the changelog fragment under docstrfmt's 120-column wrap.

Assisted-by: ClaudeCode:claude-opus-4.8
Benchmarks showed the streaming sdist scan was a no-op: tarfile's 'r:*'
mode caches every member during iteration, so time and peak memory were
identical to getmembers(). Revert it.

The memoization keeps its win (~5x on diamond-heavy dependency trees) in
a smaller form: marker check inlined as before, memo checked first.

Assisted-by: ClaudeCode:claude-fable-5
@henryiii
henryiii force-pushed the perf/dependency-check-and-sdist-scan branch from 21bf864 to f1e312e Compare July 7, 2026 19:36
@henryiii henryiii changed the title perf: memoize check_dependency and stream sdist validation perf: memoize check_dependency Jul 7, 2026
@henryiii
henryiii marked this pull request as ready for review July 7, 2026 19:44

@gaborbernat gaborbernat left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@gaborbernat
gaborbernat merged commit 820d4e5 into pypa:main Jul 7, 2026
59 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants