Repository navigation
[fix][ml] Tolerate concurrent creation of the managed ledger z-node - #26247
Merged
Merged
Conversation
MetaStoreImpl.getManagedLedgerInfo(createIfMissing = true) reads the managed ledger z-node and, when it is missing, creates it with put(path, new byte[0], Optional.of(-1L)). If another party creates the same z-node in between, the metadata store returns BadVersionException and the managed ledger initialization fails, which fences the ManagedLedgerImpl and fails the topic load. AdminResource#tryCreatePartitionAsync creates exactly the same z-node and already treats BadVersionException as "already created". The two paths race whenever a topic is loaded while its partitions are being created, for example when a replicator producer loads a partition on the remote cluster while updatePartitionedTopic runs there. Handle BadVersionException from the create by reading the z-node back: BadVersion means the z-node exists now, which is what createIfMissing asked for, and the re-read also returns the correct Stat for the subsequent compare-and-swap updates.
lhotari
added a commit
that referenced
this pull request
Aug 4, 2026
nodece
pushed a commit
to ascentstream/pulsar
that referenced
this pull request
Aug 28, 2026
…pache#26247) (cherry picked from commit d54d83d) (cherry picked from commit cd88920)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
MetaStoreImpl.getManagedLedgerInfo(createIfMissing = true)reads the managed ledger z-node and, when it is missing, creates it withput(path, new byte[0], Optional.of(-1L)). If another party creates the same z-node between that read and that write, the metadata store returnsBadVersionException, and the current code fails the callback.ManagedLedgerImplthen moves to theFencedstate and the topic load fails.AdminResource#tryCreatePartitionAsynccreates exactly the same z-node, and it already treatsBadVersionExceptionas "partition is already created" and continues. So the two paths race, and only one of them tolerates losing.The race is hit whenever a topic is loaded while its partitions are being created. It showed up as a flaky
OneWayReplicatorUsingGlobalZKTest.testPartitionedTopicWithTopicPolicyAndNoReplicationClustersfailure (branch-4.0 CI run), whereadmin.topics().updatePartitionedTopic(topic, 3, false)failed with:The broker log of the remote cluster shows the two writers colliding:
The replication producer loading
partition-0on the remote cluster collides withupdatePartitionedTopic's remote leg callingtryCreatePartitionsAsync. This is not specific to the test topology: any concurrent "create the partitions" and "load the topic" pair can hit it.Modifications
MetaStoreImpl.getManagedLedgerInfo: when the create-if-missingputfails withBadVersionException, read the z-node back instead of failing the callback.BadVersionfrom aputwith expected version-1means the z-node exists now, which is exactly whatcreateIfMissingasked for. The re-read also yields the correctStat, which the managed ledger needs for its subsequent compare-and-swap updates.The re-read is done with
createIfMissing = false, so there is no retry loop: if the z-node is gone again by then, the operation fails withMetadataNotFoundExceptionas it would for any other missing managed ledger.Verifying this change
This change added tests and can be verified as follows:
MetaStoreImplTest.createMLNodeConcurrentlyinjects the race deterministically: a spied metadata store creates the z-node behind the caller's back while still reporting it as missing, so the following create hits an already existing z-node. The test fails without the fix with the sameManagedLedgerException$BadVersionExceptionseen in CI, and passes with it.managed-ledgermodule test suite passes locally (721 tests, 0 failures).Does this pull request potentially affect one of the following parts:
Documentation
docdoc-requireddoc-not-neededdoc-completeMatching PR in forked repository
PR in forked repository: N/A (fix verified locally; apache/pulsar CI will run on this PR)