Skip to content

[fix][ml] Tolerate concurrent creation of the managed ledger z-node - #26247

Merged
nodece merged 1 commit into
apache:masterfrom
lhotari:lh-fix-ml-concurrent-znode-create
Jul 27, 2026
Merged

nodece merged 1 commit into
apache:masterfrom
lhotari:lh-fix-ml-concurrent-znode-create

Conversation

@lhotari

@lhotari lhotari commented Jul 26, 2026

Copy link
Copy Markdown
Member

Motivation

MetaStoreImpl.getManagedLedgerInfo(createIfMissing = true) reads the managed ledger z-node and, when it is missing, creates it with put(path, new byte[0], Optional.of(-1L)). If another party creates the same z-node between that read and that write, the metadata store returns BadVersionException, and the current code fails the callback. ManagedLedgerImpl then moves to the Fenced state and the topic load fails.

AdminResource#tryCreatePartitionAsync creates exactly the same z-node, and it already treats BadVersionException as "partition is already created" and continues. So the two paths race, and only one of them tolerates losing.

The race is hit whenever a topic is loaded while its partitions are being created. It showed up as a flaky OneWayReplicatorUsingGlobalZKTest.testPartitionedTopicWithTopicPolicyAndNoReplicationClusters failure (branch-4.0 CI run), where admin.topics().updatePartitionedTopic(topic, 3, false) failed with:

org.apache.pulsar.client.admin.PulsarAdminException$ServerSideErrorException:
Message: org.apache.bookkeeper.mledger.ManagedLedgerException: ...BadVersionException:
  KeeperErrorCode = BadVersion for /managed-ledgers/public/default/persistent/tp_...-partition-0
	at org.apache.pulsar.broker.service.BrokerService$2.openLedgerFailed(BrokerService.java:1953)
	at org.apache.bookkeeper.mledger.impl.ManagedLedgerFactoryImpl$2.initializeFailed(...)
	at org.apache.bookkeeper.mledger.impl.MetaStoreImpl.lambda$getManagedLedgerInfo$2(MetaStoreImpl.java:129)

The broker log of the remote cluster shows the two writers colliding:

51,081  [pulsar.repl.r1-->r2] Creating producer on ...-partition-0   -> topic load starts on r2
51,089  GET /admin/v2/persistent/public/default                       -> internalUpdatePartitionedTopicAsync on r2
51,090  Opening managed ledger ...-partition-0                        -> read reports the z-node as missing
51,092  Creating '/managed-ledgers/...-partition-0'                   -> put(..., -1) loses the race
51,097  Moving to Fenced state / Failed to initialize managed ledger: BadVersion
51,098  Failed to get stats for ...-partition-0 -> HTTP 500 -> updatePartitionedTopic fails

The replication producer loading partition-0 on the remote cluster collides with updatePartitionedTopic's remote leg calling tryCreatePartitionsAsync. This is not specific to the test topology: any concurrent "create the partitions" and "load the topic" pair can hit it.

Modifications

  • MetaStoreImpl.getManagedLedgerInfo: when the create-if-missing put fails with BadVersionException, read the z-node back instead of failing the callback. BadVersion from a put with expected version -1 means the z-node exists now, which is exactly what createIfMissing asked for. The re-read also yields the correct Stat, which the managed ledger needs for its subsequent compare-and-swap updates.

The re-read is done with createIfMissing = false, so there is no retry loop: if the z-node is gone again by then, the operation fails with MetadataNotFoundException as it would for any other missing managed ledger.

Verifying this change

This change added tests and can be verified as follows:

  • MetaStoreImplTest.createMLNodeConcurrently injects the race deterministically: a spied metadata store creates the z-node behind the caller's back while still reporting it as missing, so the following create hits an already existing z-node. The test fails without the fix with the same ManagedLedgerException$BadVersionException seen in CI, and passes with it.
  • The full managed-ledger module test suite passes locally (721 tests, 0 failures).

Does this pull request potentially affect one of the following parts:

  • Dependencies (add or upgrade a dependency)
  • The public API
  • The schema
  • The default values of configurations
  • The threading model
  • The binary protocol
  • The REST endpoints
  • The admin CLI options
  • The metrics
  • Anything that affects deployment

Documentation

  • doc
  • doc-required
  • doc-not-needed
  • doc-complete

Matching PR in forked repository

PR in forked repository: N/A (fix verified locally; apache/pulsar CI will run on this PR)

MetaStoreImpl.getManagedLedgerInfo(createIfMissing = true) reads the
managed ledger z-node and, when it is missing, creates it with
put(path, new byte[0], Optional.of(-1L)). If another party creates the
same z-node in between, the metadata store returns BadVersionException
and the managed ledger initialization fails, which fences the
ManagedLedgerImpl and fails the topic load.

AdminResource#tryCreatePartitionAsync creates exactly the same z-node
and already treats BadVersionException as "already created". The two
paths race whenever a topic is loaded while its partitions are being
created, for example when a replicator producer loads a partition on
the remote cluster while updatePartitionedTopic runs there.

Handle BadVersionException from the create by reading the z-node back:
BadVersion means the z-node exists now, which is what createIfMissing
asked for, and the re-read also returns the correct Stat for the
subsequent compare-and-swap updates.

@dao-jun dao-jun left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@nodece
nodece merged commit d54d83d into apache:master Jul 27, 2026
44 checks passed
@lhotari lhotari added this to the 5.0.0-M2 milestone Jul 27, 2026
lhotari added a commit that referenced this pull request Aug 4, 2026
lhotari added a commit that referenced this pull request Aug 4, 2026
nodece pushed a commit to ascentstream/pulsar that referenced this pull request Aug 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants