Skip to content

fix(modelartifacts): reuse committed sibling files for narrowed allow_patterns - #11484

Merged
mudler-agent merged 1 commit into
mudler:masterfrom
SuperMarioYL:fix/modelartifacts-sibling-reuse-11047
Sep 27, 2026
Merged

mudler-agent merged 1 commit into
mudler:masterfrom
SuperMarioYL:fix/modelartifacts-sibling-reuse-11047

Conversation

@SuperMarioYL

Copy link
Copy Markdown
Contributor

Description

Fixes #11047.

A request with narrower allow_patterns hashes to a different CacheKey than an already-committed broader sibling, so committedResult (the offline fast path) misses and materializeLocked re-fetches files the sibling already holds β€” re-downloading tens of GB on a narrowed request even though the model is already on disk.

This implements the "narrower shape" proposed by @Dennisadira in #11047:

  • CacheKey is left untouched β€” the artifact identity and the offline fast path are unchanged.
  • During materializeLocked's staging loop, after the own-tree reuseMaterializedFile miss, the code consults committed sibling trees for the same Source (type + endpoint + repo + revision, possibly different allow_patterns/ignore_patterns).
  • Each candidate file is re-verified through the existing verifyDownloadedFile (full SHA-256 + git SHA-1, never size-only) and hard-linked into the writer's staging snapshot/ (copy fallback only on EXDEV).

Each file is matched individually against the sibling's manifest, so a broader request can never inherit a narrower sibling's gaps as if complete.

Notes for Reviewers

  • Safety property (the question raised in the issue): a broader request can never be served from a narrower committed tree as complete β€” every reused file is full-SHA-256-verified via verifyDownloadedFile, and any missing file falls through to a normal fetch. TestBroaderRequestDoesNotServeNarrowSiblingAsComplete proves it.
  • CacheKey (pkg/modelartifacts/path.go) and committedResult are unchanged β€” no identity or offline-fast-path behavior change.
  • Disk-neutral in the common case (hard-link); a byte copy is only used when os.Link fails with EXDEV (cross-device).
  • Purely additive: no existing reuse/resume or download path is removed.

Testing

  • go test ./pkg/modelartifacts/...
  • go vet ./pkg/modelartifacts/...

New tests in pkg/modelartifacts/materializer_sibling_reuse_test.go:

  • TestReuseFromCommittedSibling_NarrowAllowPatterns β€” commits a broad tree (full file set), then Ensures a strict-subset request (narrower allow_patterns) through a recording stub snapshot resolver; asserts the sibling-present files are hard-linked (os.SameFile) and the resolver records zero fetches for them. Fails on master (re-fetches).
  • TestBroaderRequestDoesNotServeNarrowSiblingAsComplete β€” commits a narrow sibling (subset of files), then a broader request; asserts the missing file is fetched and the full manifest file-set is committed.

Signed commits

  • Yes, I signed my commits.
  • Documentation updated (docs/content/) for user-facing changes, or not applicable

Refs #11047

@SuperMarioYL
SuperMarioYL force-pushed the fix/modelartifacts-sibling-reuse-11047 branch 2 times, most recently from 0674240 to 60550e3 Compare September 1, 2026 19:21
@SuperMarioYL

Copy link
Copy Markdown
Contributor Author

Thanks for the review β€” the hoisting suggestion (read the sibling manifest set once before the staging loop instead of per-file rescans) makes sense and I'll push that follow-up shortly.

On the DCO blocker: commit 60550e32 ("test(modelartifacts): use the Ginkgo suite") was pushed to this branch by @localai-org-maint-bot via maintainer edits, and it's that commit β€” not the original fix β€” that carries no Signed-off-by. I can't legitimately sign off a commit I didn't author. Could the bot re-push it with a signoff, or is there another way you'd like it handled (happy to rebase accordingly if a maintainer prefers)?

@SuperMarioYL
SuperMarioYL dismissed a stale review via f208d68 September 11, 2026 01:42
@SuperMarioYL

Copy link
Copy Markdown
Contributor Author

Perf follow-up pushed in f208d68: loadSiblingCandidates now reads and source-matches the sibling manifest set once, before the staging loop (files indexed by path), and the per-file lookup only verifies and hard-links against those preloaded candidates β€” the 20-sibling Γ— 300-file case goes from ~6000 manifest reads and JSON parses to one pass. Match order (sibling directory order, then manifest order), the path+size check, the full SHA-256 re-verification, and the EXDEV copy fallback are unchanged; the modelartifacts suite (including both sibling-reuse specs) is green locally.

Still waiting on the DCO path for 60550e32 per my comment above.

@SuperMarioYL
SuperMarioYL force-pushed the fix/modelartifacts-sibling-reuse-11047 branch from f208d68 to 3b032cb Compare September 11, 2026 02:22
@SuperMarioYL

Copy link
Copy Markdown
Contributor Author

Quick note on the remaining DCO check: my commits on this branch are now signed off, but 60550e3 ("test(modelartifacts): use the Ginkgo suite", pushed into this branch by localai-org-maint-bot on 09-01) has no Signed-off-by, and I can't certify it on the bot's behalf. Could a maintainer re-push that change with a sign-off (or advise on the usual handling here)? Everything else on the PR is ready.

…_patterns

A request with narrower allow_patterns hashes to a different CacheKey
than an already-committed broader sibling, so committedResult misses and
materializeLocked re-fetches files the sibling already holds. After the
own-tree reuseMaterializedFile miss, consult committed sibling trees
for the same Source (type+endpoint+repo+revision), re-verify the file
through verifyDownloadedFile (full SHA-256, never size-only), and
hard-link it into the writer's staging snapshot (copy fallback only on
EXDEV). Each file is matched individually against the sibling's
manifest, so a broader request can never inherit a narrower sibling's
gaps as if complete.

The sibling manifest set is loaded and source-matched once per
materialization (files indexed by path) instead of once per staged
file, so a models volume with 20 committed artifacts and a 300-file
snapshot does one manifest pass rather than ~6000 reads and JSON
parses. The sibling-reuse behavior cases live in the package's
registered Ginkgo suite so repository test conventions apply.

Refs mudler#11047

Signed-off-by: supermario_leo <leo.stack@outlook.com>
@SuperMarioYL
SuperMarioYL force-pushed the fix/modelartifacts-sibling-reuse-11047 branch from 3b032cb to afc2b52 Compare September 13, 2026 02:16
@SuperMarioYL

Copy link
Copy Markdown
Contributor Author

Done β€” the branch is now a single squashed commit (afc2b52) authored and signed off under my identity, so the DCO has one trailer to validate and 60550e3 no longer exists as a separate unsigned commit. The tree is unchanged: the squashed diff is byte-identical to the previous three-commit diff, and go test ./pkg/modelartifacts/... passes locally (2.7s, includes both sibling-reuse specs). Once the DCO re-evaluates green, the PR should be unblocked for CI approval.

@mudler-agent mudler-agent left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed, LGTM.

@mudler-agent
mudler-agent merged commit b9634e0 into mudler:master Sep 27, 2026
68 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants