Skip to content

[improve] Wire Set<Option> through backend hooks + Oxia PartitionKey - #25723

Merged
merlimat merged 1 commit into
apache:masterfrom
merlimat:mmerli/metadata-partition-key
May 8, 2026
Merged

merlimat merged 1 commit into
apache:masterfrom
merlimat:mmerli/metadata-partition-key

Conversation

@merlimat

@merlimat merlimat commented May 8, 2026

Copy link
Copy Markdown
Contributor

Motivation

Follow-up to #25710 (the additive Set API). That PR landed the public-API surface and the canonical Set<Option> form on every read/write method, but the backend-side hooks on AbstractMetadataStore (storeGet, storeDelete, getChildrenFromStore, existsFromStore, storeScanChildren, storeFindByIndex) still took only path/version — opts were dropped at the AbstractMetadataStore boundary. That meant Option.PartitionKey, the routing hint sharded backends like Oxia need, never reached the client.

This PR plumbs Set<Option> end-to-end and wires OxiaMetadataStore to consume Option.PartitionKey natively.

Modifications

Backend hooks: AbstractMetadataStore.storeGet, storeDelete, getChildrenFromStore, existsFromStore, storeScanChildren, and storeFindByIndex all take Set<Option>. Backends override the new signatures; opts ignored where not used (LocalMemory, RocksDB).

ZK batching layer: OpGet, OpDelete, OpGetChildren each carry the opts so the batch flusher has them available; AbstractBatchedMetadataStore plumbs opts into the Op constructors.

Oxia: OxiaMetadataStore extracts PartitionKey via OptionsHelper.partitionKey and attaches the matching client option (GetOption.PartitionKey / PutOption.PartitionKey / DeleteOption.PartitionKey / ListOption.PartitionKey / RangeScanOption.PartitionKey) to every Oxia call. The two storePut overloads collapse into one.

API completeness: MetadataStore.deleteRecursive and deleteIfExists got Set<Option> overloads to round out the surface. AbstractMetadataStore.deleteRecursive threads opts through every recursive child + the final deleteIfExists so PartitionKey routing applies to the whole subtree.

Wrappers (DualMetadataStore, FaultInjectionMetadataStore): updated to the new signatures.

Verifying this change

New OxiaPartitionKeyTest runs against a 3-shard Oxia cluster and verifies put/get/delete/getChildren round-trips with Option.PartitionKey.

The test deliberately asserts only the round-trip — asserting that a record routed to shard(K) is unreachable from shard(J) is statistically flaky on 3 shards because half the (K,J) pairs collide. A stronger correctness assertion would need access to Oxia internals.

Does this pull request potentially affect one of the following parts:

  • The public MetadataStore API — strictly additive: new abstract Set<Option> overloads added for deleteRecursive and deleteIfExists, legacy no-opts forms become defaults that forward. Existing callers and external implementations are unaffected.
  • The backend-side hooks on AbstractMetadataStore — these are protected, used only by the in-tree backends, and all in-tree backends are updated in this PR.

Matching PR in forked repository

PR in forked repository: https://github.com/merlimat/pulsar/pull/new/mmerli/metadata-partition-key

…re Oxia PartitionKey

Continue the Set<Option> flip down into the backend layer so backends
receive routing/hint options on every operation, and wire OxiaMetadataStore
to consume Option.PartitionKey natively.

API surface gains canonical Set<Option> overloads for the remaining methods:
- MetadataStore.deleteRecursive(path, opts)
- MetadataStore.deleteIfExists(path, version, opts)

AbstractMetadataStore backend hooks all flipped to Set<Option>:
- storeGet, storeDelete, getChildrenFromStore, existsFromStore
- storeScanChildren, storeFindByIndex

LocalMemory, RocksDB, ZK (via AbstractBatchedMetadataStore + OpGet/OpDelete/
OpGetChildren), and Oxia updated to the new signatures. OxiaMetadataStore
extracts PartitionKey via OptionsHelper.partitionKey and attaches the matching
Oxia option (GetOption / PutOption / DeleteOption / ListOption /
RangeScanOption) to every client call.

Tests: new OxiaPartitionKeyTest exercises put/get/delete/getChildren round
trips against a 3-shard Oxia cluster.
@merlimat merlimat changed the title [improve][metadata] Wire Set<Option> through backend hooks + Oxia PartitionKey [improve] Wire Set<Option> through backend hooks + Oxia PartitionKey May 8, 2026
@lhotari
lhotari self-requested a review May 8, 2026 17:21

@lhotari lhotari left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@merlimat
merlimat merged commit 4868900 into apache:master May 8, 2026
44 of 45 checks passed
@merlimat
merlimat deleted the mmerli/metadata-partition-key branch May 8, 2026 18:19
@lhotari lhotari added this to the 5.0.0-M1 milestone Jun 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants