Repository navigation
fix(copy): re-push referenced manifests when the destination reports them missing - #1213
Merged
TerryHowe merged 2 commits intoJul 15, 2026
Conversation
…them missing
copyGraph skips a node when dst.Exists reports it present ("skip if a rooted
sub-DAG exists"). If a previous copy was interrupted after a referenced
manifest's blobs were uploaded but before that manifest itself was committed,
the destination can report the child manifest as existing while still rejecting
any manifest that references it. The push of the parent (e.g. an image index)
then fails with MANIFEST_BLOB_UNKNOWN, and because the child is skipped on every
subsequent run the recursive copy never self-heals.
When a node push fails because the destination is missing referenced content
(MANIFEST_BLOB_UNKNOWN / BLOB_UNKNOWN / MANIFEST_UNKNOWN), re-copy the node's
manifest successors unconditionally and retry the push once. This lets an
interrupted copy recover instead of failing identically on every retry. The
retry is a no-op on the success path.
Adds TestCopyGraph_SelfHealsInterruptedIndexCopy (end-to-end reproduction) and
TestIsMissingReferencedContentError (classifier unit test).
Signed-off-by: Malcolm Morgan <Malcolm.Morgan@Microsoft.com>
Orexii
requested review from
TerryHowe,
Wwwsylvia,
sabre1041 and
shizhMSFT
as code owners
June 10, 2026 14:56
Contributor
Author
|
@TerryHowe anything needed to do on my end for this? Please let me know. Thanks |
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Backport of #1212 to the
v2branch.What this does
Makes
CopyGraphself-heal a partially-copied graph. When the destination rejects a manifest push because it is missing referenced content (MANIFEST_BLOB_UNKNOWN/BLOB_UNKNOWN/MANIFEST_UNKNOWN), the copy now re-pushes the node's manifest successors and retries the push once, instead of returning the error.Why
CopyGraphskips a node whose sub-DAG it believes already exists ("skip if a rooted sub-DAG exists"). If a prior copy was interrupted after a child manifest's blobs were uploaded but before the child manifest was committed, the destination can report the child as present (Exists/HEAD) yet reject the parent that references it. For a multi-arch image index this means the index push fails withMANIFEST_BLOB_UNKNOWN, and since the child is skipped on every run, re-running the sameoras cp --recursivenever recovers. This is the line that reaches the ORAS CLI.This was hit in production mirroring a multi-arch image (
mongo:7.0.34) between two registries; the only manual workaround was copying each child manifest by digest and re-pushing the index.How
copyGraph: wrap the node push; on a "missing referenced content" error, re-copy the manifest successors unconditionally (bypassing the existence short-circuit) and retry the push once. No-op on the success path; blob successors are not re-pushed.isMissingReferencedContentError: typed classification viaerrcode(handles both a singleerrcode.Errorand a multi-errorerrcode.Errors); no string matching.Testing
TestCopyGraph_SelfHealsInterruptedIndexCopy: end-to-end reproduction against a destination that emulates the interrupted state — fails without this change (MANIFEST_BLOB_UNKNOWN), passes with it.TestIsMissingReferencedContentError: unit test for the classifier (100% covered).go test -race ./...passes;gofmt/go vetclean; new files carry the Apache license header.The diff is identical to #1212 apart from the module import paths (
github.com/oras-project/oras-go/v3->oras.land/oras-go/v2).Relates to #1211.