Repository navigation
[improve][test] pulsar-perf consume: switch between V5 Queue and Stream consumer - #25981
Merged
Merged
Conversation
…am consumer Adds --scalable-consumer-type (-sct) Queue|Stream to the 'consume' perf command. Queue keeps the existing unordered, individually-acked QueueConsumer; Stream drives the V5 StreamConsumer (ordered, cumulative ack, broker-coordinated 1:1 segment-to-consumer assignment). Running with Stream and more consumers than segments is the handle for manually exercising scalable-topic auto-split (PIP-483). - A thin PerfConsumer adapter unifies receive/ack across the two V5 consumer APIs (acknowledge vs acknowledgeCumulative), so the poll loop and transaction path are type-agnostic. - Consumer creation refactored into subscribeAsync(); encryption-policy construction extracted into a single helper (also removes a dead, never-subscribed QueueConsumerBuilder that the old code built and discarded). - Stream consumers get a distinct consumerName per index so they're distinct group members. Tested: PerformanceConsumerArgsTest covers the new option (default Queue, long/short forms, Stream).
…sumer name Setting consumerName = <sub>-c<index> made two separate 'pulsar-perf consume' processes both register as e.g. sub-c0. The scalable-topic controller keys group membership by consumer name, so the second process was treated as a reconnect of the first and handed the same exclusively-held segment — failing with ConsumerBusyException instead of joining the group. Drop the override and let the V5 client auto-generate a unique consumer name (v5-stream-<random>). Each consumer — within one process (-n N) and across separate invocations — is now a distinct group member, so an extra consumer correctly gets an empty assignment until auto-split adds a segment for it.
lhotari
approved these changes
Jun 9, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
The
pulsar-perf consumecommand always drives the V5QueueConsumer. To manually exercise scalable-topic features — in particular auto-split (PIP-483, where a subscription with more consumers than segments should trigger a split — there needs to be a way to run it as aStreamConsumerinstead (ordered, cumulative ack, broker-coordinated 1:1 segment-to-consumer assignment).Modifications
--scalable-consumer-type/-scton theconsumecommand:Queue(default, unchanged behavior) orStream.PerfConsumeradapter unifiesreceive/ ack across the two V5 consumer APIs (acknowledgefor Queue,acknowledgeCumulativefor Stream), so the poll loop and transaction path are type-agnostic.subscribeAsync()factory; encryption-policy construction extracted into a single helper. This also removes a pre-existing deadQueueConsumerBuilderthat the old code built (with an encryption policy) and then never subscribed.Streamconsumers get a distinctconsumerNameper index so the N consumers are distinct members of the group and the controller assigns segments 1:1.Usage to drive auto-split:
Verifying this change
PerformanceConsumerArgsTestcovers the new option (defaultQueue, long/short forms,Stream).pulsar-testclientcompiles and checkstyle passes.Does this pull request potentially affect one of the following parts:
This only adds a CLI option to the
pulsar-perftool; default behavior (Queue) is unchanged.