Repository navigation
perf: memoize check_dependency - #1122
Merged
gaborbernat merged 4 commits intoJul 7, 2026
Merged
Conversation
check_dependency recursively verified a requirement and its transitive deps with only a cycle guard, so a diamond dependency graph re-walked each shared subtree once per path (exponential on dense graphs, visible with --no-isolation dependency checks). Thread a ``seen`` memo of requirement strings already verified as satisfied through an internal helper. Marker evaluation depends on parent_extras, but past the marker gate the outcome depends only on the requirement itself, so memoising past that gate by the normalised string is correct; only fully satisfied subtrees are recorded so unmet requirements are still reported per chain. _validate_sdist_archive called tar.getmembers() (materialising every TarInfo) and iterated it twice. Stream the archive once, accumulating top-level segments and which of them carry a <top>/PKG-INFO regular file, for memory/CPU proportional to a single pass on large sdists. Behaviour and error messages are unchanged. Assisted-by: ClaudeCode:claude-opus-4.8
Assisted-by: ClaudeCode:claude-opus-4.8
CI enforces 100% coverage including test files. Exhaust the check_dependency generator for an incompatible version and an unmet transitive dependency (existing tests stop at the first yield via ``next``), add an empty-name tar member to exercise the streaming skip, and drop a dead not-found branch in the diamond test mock. Shorten the changelog fragment under docstrfmt's 120-column wrap. Assisted-by: ClaudeCode:claude-opus-4.8
Benchmarks showed the streaming sdist scan was a no-op: tarfile's 'r:*' mode caches every member during iteration, so time and peak memory were identical to getmembers(). Revert it. The memoization keeps its win (~5x on diamond-heavy dependency trees) in a smaller form: marker check inlined as before, memo checked first. Assisted-by: ClaudeCode:claude-fable-5
henryiii
force-pushed
the
perf/dependency-check-and-sdist-scan
branch
from
July 7, 2026 19:36
21bf864 to
f1e312e
Compare
henryiii
marked this pull request as ready for review
July 7, 2026 19:44
gaborbernat
approved these changes
Jul 7, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Okay, I think this speedup is pretty clean, and can be seen for large projects:
I removed the second change, as it wasn't measurably faster and was a costly change in code complexity.
🤖 Performance report 🤖
Measured both changes against
main. Bottom line: the memoization is real but only matters in unusual cases; the sdist streaming change is a measured no-op.check_dependency memoization
Results were identical (same unmet-dependency sets) in every case. The 4.3x looks impressive, but context matters: this code path only runs with
--no-isolation(isolated builds skip it,__main__.py:242), plus a tiny fixedbuild[uv]check inenv.py:308. The trees actually walked are the build backend's requirements — hatchling, setuptools, flit_core — which are small and shallow. There the saving is ~0.1 ms per build. You'd need a backend (or a third-party caller of the publicbuild.check_dependency) depending on something jupyterlab-shaped to save the ~165 ms, once, in builds that take seconds anyway.Sdist streaming validation
Zero benefit in both time and peak memory, and there's a structural reason:
tarfile.openin the defaultr:*mode caches everyTarInfoontar.membersas you iterate, so "streaming" iteration allocates exactly whatgetmembers()does. Only true stream mode (r|*) would change that — and even the absurd 100k-member case peaks at just 46 MB.My take
build's own call sites the saving is microseconds. The only argument for keeping it is thatcheck_dependencyis public API and a--no-isolationuser with an exotic backend gets protection from pathological diamond graphs — but 214 ms is the worst case I could construct with a real package, and that's hardly a problem worth guarding against.If you want the branch to earn its keep, the memoization alone (with the marker-helper refactor folded back in) is the defensible half; I wouldn't ship the sdist part.
🤖 AI text below 🤖
Part of the code review in #1116.
Two behaviour-preserving performance improvements found in review.
check_dependencymemoization (src/build/_util.py). The recursive dependency check only had a cycle guard, so a diamond dependency graph (A→B,C; B→D; C→D) re-walked each shared subtree once per path — exponential blowup on dense graphs, visible with--no-isolationdependency checks. Aseenset of requirement strings already verified as satisfied is now threaded through an internal_check_dependencyhelper (public signature and yielded tuple format unchanged). Correctness subtlety: marker evaluation depends onparent_extras, but past the marker gate the outcome depends only on the requirement itself, so the memo lives past that gate and is keyed by the normalised requirement string. Only fully satisfied subtrees are recorded, so unmet requirements are still reported once per dependency chain. Regression test added (test_check_dependency_diamond_visits_each_once) asserting each package is looked up at most once; it fails before the fix (D looked up twice)._validate_sdist_archivestreaming (src/build/__main__.py). Replacedtar.getmembers()(which materialized aTarInfofor the whole archive, then iterated it twice) with a single streaming pass over theTarFile, accumulating top-level name segments and which of them carry a<top>/PKG-INFOregular file. Memory/CPU now proportional to one pass; all error messages and behaviour (including the empty-top-leveldiscard()handling) are unchanged. Addedtest_validate_sdist_archive_ignores_empty_top_levelcovering the empty-segment edge.All existing
check_dependencyand sdist-validation tests pass unchanged.pytest,mypy, andprekare green.