Repository navigation
fix(modelartifacts): reuse committed sibling files for narrowed allow_patterns - #11484
Conversation
0674240 to
60550e3
Compare
|
Thanks for the review β the hoisting suggestion (read the sibling manifest set once before the staging loop instead of per-file rescans) makes sense and I'll push that follow-up shortly. On the DCO blocker: commit |
|
Perf follow-up pushed in f208d68: Still waiting on the DCO path for |
f208d68 to
3b032cb
Compare
|
Quick note on the remaining DCO check: my commits on this branch are now signed off, but |
β¦_patterns A request with narrower allow_patterns hashes to a different CacheKey than an already-committed broader sibling, so committedResult misses and materializeLocked re-fetches files the sibling already holds. After the own-tree reuseMaterializedFile miss, consult committed sibling trees for the same Source (type+endpoint+repo+revision), re-verify the file through verifyDownloadedFile (full SHA-256, never size-only), and hard-link it into the writer's staging snapshot (copy fallback only on EXDEV). Each file is matched individually against the sibling's manifest, so a broader request can never inherit a narrower sibling's gaps as if complete. The sibling manifest set is loaded and source-matched once per materialization (files indexed by path) instead of once per staged file, so a models volume with 20 committed artifacts and a 300-file snapshot does one manifest pass rather than ~6000 reads and JSON parses. The sibling-reuse behavior cases live in the package's registered Ginkgo suite so repository test conventions apply. Refs mudler#11047 Signed-off-by: supermario_leo <leo.stack@outlook.com>
3b032cb to
afc2b52
Compare
|
Done β the branch is now a single squashed commit ( |
Description
Fixes #11047.
A request with narrower
allow_patternshashes to a differentCacheKeythan an already-committed broader sibling, socommittedResult(the offline fast path) misses andmaterializeLockedre-fetches files the sibling already holds β re-downloading tens of GB on a narrowed request even though the model is already on disk.This implements the "narrower shape" proposed by @Dennisadira in #11047:
CacheKeyis left untouched β the artifact identity and the offline fast path are unchanged.materializeLocked's staging loop, after the own-treereuseMaterializedFilemiss, the code consults committed sibling trees for the same Source (type + endpoint + repo + revision, possibly differentallow_patterns/ignore_patterns).verifyDownloadedFile(full SHA-256 + git SHA-1, never size-only) and hard-linked into the writer's stagingsnapshot/(copy fallback only onEXDEV).Each file is matched individually against the sibling's manifest, so a broader request can never inherit a narrower sibling's gaps as if complete.
Notes for Reviewers
verifyDownloadedFile, and any missing file falls through to a normal fetch.TestBroaderRequestDoesNotServeNarrowSiblingAsCompleteproves it.CacheKey(pkg/modelartifacts/path.go) andcommittedResultare unchanged β no identity or offline-fast-path behavior change.os.Linkfails withEXDEV(cross-device).Testing
go test ./pkg/modelartifacts/...go vet ./pkg/modelartifacts/...New tests in
pkg/modelartifacts/materializer_sibling_reuse_test.go:TestReuseFromCommittedSibling_NarrowAllowPatternsβ commits a broad tree (full file set), thenEnsures a strict-subset request (narrowerallow_patterns) through a recording stub snapshot resolver; asserts the sibling-present files are hard-linked (os.SameFile) and the resolver records zero fetches for them. Fails onmaster(re-fetches).TestBroaderRequestDoesNotServeNarrowSiblingAsCompleteβ commits a narrow sibling (subset of files), then a broader request; asserts the missing file is fetched and the full manifest file-set is committed.Signed commits
Refs #11047