Skip to content

[improve][test] pulsar-perf consume: switch between V5 Queue and Stream consumer - #25981

Merged
lhotari merged 2 commits into
apache:masterfrom
merlimat:perf-consumer-stream-queue
Jun 9, 2026
Merged

lhotari merged 2 commits into
apache:masterfrom
merlimat:perf-consumer-stream-queue

Conversation

@merlimat

@merlimat merlimat commented Jun 9, 2026

Copy link
Copy Markdown
Contributor

Motivation

The pulsar-perf consume command always drives the V5 QueueConsumer. To manually exercise scalable-topic features — in particular auto-split (PIP-483, where a subscription with more consumers than segments should trigger a split — there needs to be a way to run it as a StreamConsumer instead (ordered, cumulative ack, broker-coordinated 1:1 segment-to-consumer assignment).

Modifications

  • New option --scalable-consumer-type / -sct on the consume command: Queue (default, unchanged behavior) or Stream.
  • A thin PerfConsumer adapter unifies receive / ack across the two V5 consumer APIs (acknowledge for Queue, acknowledgeCumulative for Stream), so the poll loop and transaction path are type-agnostic.
  • Consumer creation refactored into a subscribeAsync() factory; encryption-policy construction extracted into a single helper. This also removes a pre-existing dead QueueConsumerBuilder that the old code built (with an encryption policy) and then never subscribed.
  • Stream consumers get a distinct consumerName per index so the N consumers are distinct members of the group and the controller assigns segments 1:1.

Usage to drive auto-split:

pulsar-perf consume -sct Stream -n 8 topic://tenant/ns/my-scalable-topic

Verifying this change

  • PerformanceConsumerArgsTest covers the new option (default Queue, long/short forms, Stream).
  • pulsar-testclient compiles and checkstyle passes.

Does this pull request potentially affect one of the following parts:

  • Dependencies
  • The public API
  • The schema
  • The default values of configurations
  • The threading model
  • The binary protocol
  • The REST endpoints
  • The admin CLI options
  • Anything that affects deployment

This only adds a CLI option to the pulsar-perf tool; default behavior (Queue) is unchanged.

merlimat added 2 commits June 8, 2026 16:57
…am consumer

Adds --scalable-consumer-type (-sct) Queue|Stream to the 'consume' perf command. Queue keeps the existing unordered, individually-acked QueueConsumer; Stream drives the V5 StreamConsumer (ordered, cumulative ack, broker-coordinated 1:1 segment-to-consumer assignment). Running with Stream and more consumers than segments is the handle for manually exercising scalable-topic auto-split (PIP-483).

- A thin PerfConsumer adapter unifies receive/ack across the two V5 consumer APIs (acknowledge vs acknowledgeCumulative), so the poll loop and transaction path are type-agnostic.
- Consumer creation refactored into subscribeAsync(); encryption-policy construction extracted into a single helper (also removes a dead, never-subscribed QueueConsumerBuilder that the old code built and discarded).
- Stream consumers get a distinct consumerName per index so they're distinct group members.

Tested: PerformanceConsumerArgsTest covers the new option (default Queue, long/short forms, Stream).
…sumer name

Setting consumerName = <sub>-c<index> made two separate 'pulsar-perf consume' processes both register as e.g. sub-c0. The scalable-topic controller keys group membership by consumer name, so the second process was treated as a reconnect of the first and handed the same exclusively-held segment — failing with ConsumerBusyException instead of joining the group.

Drop the override and let the V5 client auto-generate a unique consumer name (v5-stream-<random>). Each consumer — within one process (-n N) and across separate invocations — is now a distinct group member, so an extra consumer correctly gets an empty assignment until auto-split adds a segment for it.
@lhotari
lhotari merged commit 36270cc into apache:master Jun 9, 2026
42 of 43 checks passed
@merlimat
merlimat deleted the perf-consumer-stream-queue branch June 9, 2026 15:28
@lhotari lhotari added this to the 5.0.0-M1 milestone Jun 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants