Skip to content

[fix][test] Fix flaky V5DagWatchAutoReconnectTest producer test - #26510

Merged
nodece merged 2 commits into
apache:masterfrom
nodece:fix-flaky-v5-dag-watch-reconnect-test
Sep 9, 2026
Merged

nodece merged 2 commits into
apache:masterfrom
nodece:fix-flaky-v5-dag-watch-reconnect-test

Conversation

@nodece

@nodece nodece commented Sep 9, 2026

Copy link
Copy Markdown
Member

Motivation

testProducerSurvivesDagWatchConnectionDrop in V5DagWatchAutoReconnectTest is flaky: the exact-set assertion intermittently fails with producer must keep sending after DAG watch reconnect kicks in.

The test force-closes the producer's DAG-watch channel, but with the default connectionsPerBroker=1 the v5 client pools a single connection per broker, so that channel is the shared connection for the whole client. The consumer's segment consumers ride the same connection and reconnect alongside the producer; around that window the broker can redeliver first-batch messages (at-least-once semantics), which pollutes the second drain and makes the exact-set assertion timing-dependent.

Modifications

  • Create the consumer on its own dedicated client (newV5Client()) so the force-close is scoped to the producer's connections only. The producer's segment producers still drop and reconnect (sends issued in the window are queued and flushed once the connection is re-established), but the consumer stays connected — removing the reconnect/redelivery race and making the assertion deterministic.

Verifying this change

This change updates an existing test and can be verified by running:

./gradlew :pulsar-broker:test --tests "org.apache.pulsar.client.api.v5.V5DagWatchAutoReconnectTest"
  • The change passes the CI checks locally (BUILD SUCCESSFUL, :pulsar-broker:checkstyleTest clean).

Does this pull request potentially affect one of the following parts:

  • Dependencies (add or upgrade a dependency)
  • The public API
  • The schema
  • The default values of configurations
  • The threading model
  • The binary protocol
  • The REST endpoints
  • The admin CLI options
  • The metrics
  • Anything that affects deployment

Documentation

  • doc-required
  • doc-not-needed
  • doc
  • doc-omitted

### Motivation

testProducerSurvivesDagWatchConnectionDrop in V5DagWatchAutoReconnectTest is
flaky: the exact-set assertion intermittently fails with "producer must keep
sending after DAG watch reconnect kicks in".

The test force-closes the producer's DAG-watch channel, but with the default
connectionsPerBroker=1 the v5 client pools a single connection per broker, so
that channel is the shared connection for the whole client. The consumer's
segment consumers ride the same connection and reconnect alongside the
producer; around that window the broker can redeliver first-batch messages
(at-least-once), which pollutes the second drain and makes the exact-set
assertion timing-dependent.

### Modifications

- Create the consumer on its own dedicated client (newV5Client()) so the
  force-close is scoped to the producer's connections. The producer's segment
  producers still drop and reconnect (sends queue and flush), but the consumer
  stays connected, removing the reconnect/redelivery race and making the
  assertion deterministic.

Assisted-by: Codex (OpenAI)
…nectTest

Document why the second batch still succeeds: the force-close also drops the
shared client connection, so the per-segment v4 producers reconnect and sends
issued in the window are queued and flushed once the connection is
re-established.

Assisted-by: Codex (OpenAI)
@nodece
nodece merged commit 9dced41 into apache:master Sep 9, 2026
42 checks passed
@nodece nodece added this to the 5.0.0-M2 milestone Sep 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants