Skip to content

Update Test Server Runner (Major Version) - #1617

Merged
scottf merged 2 commits into
mainfrom
update-server-runner
Aug 20, 2026
Merged

scottf merged 2 commits into
mainfrom
update-server-runner

Conversation

@scottf

@scottf scottf commented Aug 20, 2026

Copy link
Copy Markdown
Contributor

Additionally

  • fixed a flaky test
  • improved test configuration files

Additionally
- fixed a flaky test
- improved test configuration files
@synadia-claude-reviewer

synadia-claude-reviewer Bot commented Aug 20, 2026 •

Copy link
Copy Markdown

Claude finished @scottf's task in 4m 56s โ€”โ€” View job


PR Review: Update Test Server Runner (Major Version)

Tasks:

  • Gather context and understand PR scope
  • Read PR diff and changed files
  • Analyze for correctness, security, and performance issues
  • Check thread safety and synchronization
  • Check exception handling patterns
  • Review JetStream API usage
  • Post inline comments for issues found
  • Write final review summary

Review Summary

The PR is a clean mechanical migration from jnats-server-runner v3 to v4, plus two substantive improvements. Overall the changes are correct. I found two issues worth addressing before merging.

Issues

1. SNAPSHOT dependency (inline comment on build.gradle)

testImplementation 'io.nats:jnats-server-runner:4.0.1-SNAPSHOT'

SNAPSHOT artifacts are mutable โ€” the same coordinates can resolve to different bytecode across builds and CI runs. This breaks reproducibility and makes regressions hard to bisect. Should be replaced with a final release version before landing on main.

2. Privileged WS ports in static config files (inline comment on AuthTests.java)

ws_operator.conf hardcodes ws { port: 80 } and wss_operator.conf hardcodes ws { port: 443 }. Both are privileged ports on Linux. Previously these tests accidentally connected to the NATS port via a ws:// scheme (old getLocalhostUri("ws") used getPort() = NATS port), which masked the problem. Now that the calls correctly use ts.getNonNatsPort(), the NATS server must actually bind to port 80/443 โ€” which fails without root or CAP_NET_BIND_SERVICE.

ws_operator_reconnect.conf (newly added by this PR) already solves this correctly via the <ws> named-port placeholder. The static operator config files should be updated the same way.

Positive changes worth noting

  • Flaky test fix (testReconnectOverOrdered): Reusing a single NatsTestServer instance with ts.start() / ts.close() across server restarts preserves the JetStream file-store directory, which is exactly what a file-stream reconnect test needs. The previous approach of creating new server instances lost the store on each restart.

  • Increased validation robustness: Bumping from 5 tries with short delays to 10 tries ร— 200 ms timeout per attempt gives slower CI environments more time to detect a started server. The same improvement in LongRunningNatsTestServer (15 tries ร— 200 ms) is consistent.

  • testReconnectToSecondServerFromInfo: The added assertNotNull(nc.getConnectedUrl()) before the equality check gives a more actionable failure message if the connection URL is unexpectedly null.

  • ws_operator_reconnect.conf: New config file using the v4 named-port placeholder syntax (<p> for NATS, <ws> for WebSocket) is the right pattern; the static operator configs should follow suit (see issue Revamped POM.ย #2 above).

  • TestBase.runInJsHubLeaf: Adding "port: " + hubPort to hubInserts fixes a latent asymmetry โ€” leafInserts already had this entry; hubInserts did not.

Comment thread build.gradle Outdated

testImplementation 'org.junit.jupiter:junit-jupiter:5.14.1'
testImplementation 'io.nats:jnats-server-runner:3.0.1'
testImplementation 'io.nats:jnats-server-runner:4.0.1-SNAPSHOT'

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SNAPSHOT dependency makes builds non-reproducible โ€” the artifact can change under CI without a version bump. Please resolve to a final release (4.0.1 or whatever the stable tag is) before merging to main.

@scottf scottf Aug 20, 2026 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed.

@scottf

scottf commented Aug 20, 2026

Copy link
Copy Markdown
Contributor Author

@claude. Please re-review considering comments

@synadia-claude-reviewer

synadia-claude-reviewer Bot commented Aug 20, 2026 •

Copy link
Copy Markdown

Claude finished @scottf's task in 3m 23s โ€”โ€” View job


Re-Review: Update Test Server Runner (Major Version)

Tasks:

  • Understand request โ€” re-review considering inline comment responses
  • Read ws_operator.conf and wss_operator.conf to understand port handling
  • Read NatsTestServer.java and ws.conf to understand v4 port replacement mechanism
  • Verify build.gradle SNAPSHOT fix
  • Analyze AuthTests.java changes in context of Scott's clarification
  • Write updated review summary

Updated Review

Issue 1: SNAPSHOT dependency โ€” Resolved โœ“

Confirmed fixed. build.gradle:40 now reads:

testImplementation 'io.nats:jnats-server-runner:4.0.1'

No SNAPSHOT suffix. This is correct.

Issue 2: Privileged WS ports โ€” Original analysis was wrong

Scott's correction is accurate. Reading ws.conf makes the v4 runner's model clear:

# "<ws>" in the port line will be replaced with a generated port
# and will be retrievable via NatsTestServer.getPort("ws")

The runner pre-processes the config and replaces port placeholders before handing the config to the NATS server. In ws_operator.conf, port 80 serves as a generic numeric placeholder that the runner replaces with a dynamically assigned port. <ws> is the named placeholder alternative โ€” both result in a dynamic port, but <ws> adds named retrievability. getNonNatsPort() returns whichever was assigned, so neither form actually attempts to bind a privileged port. My original concern was unfounded.

Minor observation: stale comment in ws.conf

src/test/resources/ws.conf line 3 still references the v3 API:

# and will be retrievable via NatsTestServer.getPort("ws")

The current v4 API is getNonNatsPort(). This doesn't affect test execution but could mislead someone writing new tests. Not a blocker, but worth a follow-up cleanup.

Other changes look correct

  • _testReconnect refactored to accept a ws flag and pre-allocate both natsPort and wsPort before constructing the test server โ€” this is the right pattern for reconnect tests where the port must stay stable across server restarts.
  • assertNotNull(nc.getConnectedUrl()) added before the endsWith check in testReconnectToSecondServerFromInfo โ€” gives a clearer failure message.
  • All getPort() โ†’ getNatsPort() and getPort("ws") โ†’ getNonNatsPort() renames are mechanical and correct.
  • ws_operator_reconnect.conf using <p> / <ws> named placeholders is consistent with the v4 runner's preferred style.

Summary

Both issues from the initial review have been addressed โ€” the SNAPSHOT is resolved, and the port concern was based on a misunderstanding of the runner's behavior. The PR is a clean migration with two solid improvements (flaky test fix, hub port fix in runInJsHubLeaf). The ws.conf comment is a minor doc debt but not a merge blocker.

assertFalse(ses.isShutdown());

options = standardOptionsBuilder(server.getNatsLocalhostUri())
options = standardOptionsBuilder(server.getURI())

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why are these renaming the opposite way to the others?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added the getUri because if I didn't, a lot more files would have been touched. getNatsLocalhostUri is superseded by a change in the tool library.

String[] servers = new String[]{
ts1.getNatsLocalhostUri(),
ts2.getNatsLocalhostUri()
ts1.getURI(),

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Again here, why renaming the other way, this time definitely methods on the test server?

server(ts.getURI()).errorListener(listener).build())) {
NatsConnection nc = (NatsConnection) Nats.connect(
new Options.Builder().
server(ts.getURI())

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems a candidate for the rename too?

NatsTestServer.builder()
.configFilePath("src/test/resources/tls_first.conf")
.connectValidateTlsFirstMode())
.skipConnectValidate())

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why skip validation? Also line 467.

And then some renames in the opposite direction, as previously noted.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Validate is done by the tool that starts the server. But it assumes a regular server mode not tls first. this is a shortcoming in the tool. So the test will fail if the tool tries to validate the server connection.


standardRequestReply(Options.builder().
server(getLocalhostUri("ws", ts.getPort("ws")))
server(getLocalhostUri("ws", ts.getNonNatsPort()))

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This getNonNatsPort seems to me an odd naming choice, since NATS supports MQTT too and might support others in future. Seems better to use something named for WS, or stick to something which passes the protocol in as a parameter as before.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The server runner has had use in handling regular connections (nats port) and websocket connections (non nats port). But without adding additional behavior to specifically understand every single block in the server config to the server runner, non nats port covers whatever type of other port.

@scottf
scottf merged commit 103c365 into main Aug 20, 2026
35 checks passed
@scottf
scottf deleted the update-server-runner branch August 20, 2026 12:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants