Skip to content

fix(microservices): let the mqtt client retry after a failed connect - #17891

Merged
kamilmysliwiec merged 2 commits into
nestjs:masterfrom
hktitof:fix/microservices-mqtt-retry-after-failed-connect
Sep 28, 2026
Merged

kamilmysliwiec merged 2 commits into
nestjs:masterfrom
hktitof:fix/microservices-mqtt-retry-after-failed-connect

Conversation

@hktitof

@hktitof hktitof commented Sep 27, 2026

Copy link
Copy Markdown
Contributor

Context: sibling to the redis, rmq, kafka and nats client retry fixes that reset a cached failed attempt so a later connect() tries again. This does the same for ClientMqtt, which still caches the rejection when mqtt gives up after the first failure

PR Checklist

Please check if your PR fulfills the following requirements:

PR Type

What kind of change does this PR introduce?

  • Bugfix
  • Feature
  • Code style update (formatting, local variables)
  • Refactoring (no functional changes, no api changes)
  • Build related changes
  • CI related changes
  • Other... Please describe:

What is the current behavior?

when reconnectPeriod is 0, mqtt does not retry after the first failed connection, it emits error and close and stays closed on the same client. connect() caches the rejected attempt in connectionPromise and keeps mqttClient, so every later connect() call sees mqttClient set and returns that same rejection instead of trying again. the only way out is a close() first

this reproduces with the real mqtt driver on master against a dead port. the first connect rejects, then the second connect creates no new client and rejects with the same cached error

PROBE first reject: Error: connect ECONNREFUSED 127.0.0.1:65534
PROBE after first: created= 1 mqttClient set= true promise set= true
PROBE second connect: created= 1 (no new client)
PROBE second reject: Error: connect ECONNREFUSED 127.0.0.1:65534
PROBE events: ["error:ECONNREFUSED","close"] reconnecting= false

Issue Number: N/A

Steps to reproduce

  1. run npx vitest run packages/microservices/test/client/client-mqtt.spec.ts on master with the fix reverted (the tests in the connect after a failed attempt block)
  2. Expected: the next connect() starts over with a new client
  3. Actual (raw output on untouched master):
Test Files  1 failed (1)
     Tests  2 failed | 48 passed (50)

What is the new behavior?

a failed attempt only drops the client when mqtt will not reconnect on its own, and a closed client that gave up reconnecting is dropped too, so the next connect() starts over with a new client. when reconnectPeriod is set, mqtt keeps retrying on the same client and nothing is dropped, so the background reconnect still resolves the promise on connect. the same probe now creates a new client on the second call

PROBE after first: created= 1 mqttClient set= false promise set= false
PROBE second connect: created= 2 (new client)
PROBE after second: created= 2 mqttClient set= false

Does this PR introduce a breaking change?

  • Yes
  • No

Other information

the redis, rmq, kafka and nats clients all reset a failed attempt already, mqtt was the one client left caching it

when reconnectPeriod is 0, mqtt gives up after the first failed connection
and stays closed on the same client, so the rejected attempt stays cached and
every later connect() call returns that same rejection instead of trying
again, the same as the redis, rmq, kafka and nats clients got in the sibling
fixes

drop the client when a failed attempt cannot reconnect on its own and when a
closed client gave up reconnecting, so the next connect() call starts over
with a new client

@micalevisk micalevisk left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for closing the gap the kafka fix (#17816) left open, and for the probe in the description: it reproduces with mqtt@5.16.0 exactly as described, and the two new "drop" specs fail on master and pass here.

I ran this against a real checkout: npx vitest run packages/microservices (63 files, 932 tests green, the 2 new specs red once the base client-mqtt.ts is swapped in), npm run build, npm run lint:ci, prettier on both files. The integration/ MQTT suites need the mosquitto container from docker-compose.yml and were not run. I also drove the real driver against a closed port to check the event order this relies on: with reconnectPeriod: 0, error fires (with reconnecting === false) one loop turn before close, so the .catch branch discards first and the close listener finds the client already gone; with reconnectPeriod > 0, mqtt.js's own close handler runs _setupReconnect() before ours, so reconnecting is already true when registerCloseListener looks at it and the client is kept. The guards hold.

One blocking issue, though. The close-listener path is new behaviour (a connection that was established and then dropped with reconnectPeriod: 0 now gets replaced on the next connect()), but discardClient() only resets mqttClient and connectionPromise. isInitialConnection stays true, so the replacement client never gets the message listener; subscriptionsCount keeps the counts of the dead connection, so publish() skips subscribe() on the new client; and the requests still in routingMap are never failed. Every send() after the "recovery" hangs while the status reads CONNECTED. The redis fix in this same series (#17828, discardClients()) resets wasInitialConnectionSuccessful for exactly this reason, and close() here already does the same reset. Details, a repro and a spec in the inline comment.

Non-blocking, but please take a look while you are in there:

  • connect() issued while close() awaits endAsync() now creates a second live client that close() then forgets (inline comment on the close listener).
  • The !client.reconnecting guard is not pinned by any spec: removing it leaves all 50 tests green (inline comment on the spec).
  • Listeners registered through ClientMqtt#on() stay on the dropped client; #17828 kept them for later clients. Fine as a follow-up (inline comment).

No docs follow-up needed. "No breaking change" is fair; the only visible change besides the retry is that unwrap() throws Not initialized after an automatic drop, same as ClientNats does since #17861.

Comment on lines +141 to 147
private discardClient(client: MqttClient) {
if (client !== this.mqttClient) {
return;
}
this.mqttClient = null;
this.connectionPromise = null;
}

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking. For the first-connect failure this is enough, but this method is also what the new close listener calls after a connection that was established and then dropped with reconnectPeriod: 0. In that case the next connect() does create a new client, as intended, but the rest of the state still describes the dead one:

  • isInitialConnection stays true, so registerConnectListener never attaches the message listener to the replacement client (client-mqtt.ts:227-230). Replies are never routed.
  • subscriptionsCount keeps the counts of the dropped connection, so publish() takes the subscriptionsCount > 0 branch and never calls subscribe() on the new client (client-mqtt.ts:354-360). The broker never delivers the replies to it either.
  • The callbacks still in routingMap are never failed, unlike in close().

Every send() after the "recovery" hangs, with the status at CONNECTED. Reproduced with fakes on this branch: after connect → one in-flight request → close → connect, the second client has listenerCount('message') === 0, subscribe is not called for the second request, and a reply emitted on the second client reaches no callback.

close() (lines 71-75) already resets exactly this, and the redis twin of this fix (#17828, discardClients()) resets wasInitialConnectionSuccessful for the same reason. On #17691 the argument for resetting isInitialConnection only in close() was that an ended client never emits connect again; here a new client will.

Suggested change
private discardClient(client: MqttClient) {
if (client !== this.mqttClient) {
return;
}
this.mqttClient = null;
this.connectionPromise = null;
}
private discardClient(client: MqttClient) {
if (client !== this.mqttClient) {
return;
}
this.mqttClient = null;
this.connectionPromise = null;
// Same reset as close(): the next client has to get the response
// listener again and subscribe to the reply channels anew, and the
// requests still waiting on the dropped connection cannot be answered.
this.isInitialConnection = false;
this.subscriptionsCount.clear();
this.handleClose();
}

And a spec for it (fails on this branch, passes with the reset), next to the other three:

it('should serve requests again after a dropped connection is replaced', async () => {
  const mqtt = new ClientMqtt({});
  const untyped: any = mqtt;
  const first = fakeClient(0);
  const second = fakeClient(0);
  for (const c of [first, second]) {
    c.subscribe = vi.fn((_channel, done) => done());
    c.publish = vi.fn();
  }
  vi.spyOn(mqtt, 'createClient')
    .mockReturnValueOnce(first)
    .mockReturnValueOnce(second);
  vi.spyOn(mqtt, 'assignPacketId' as any).mockImplementation(packet =>
    Object.assign(packet as object, { id: '1' }),
  );

  const connected = mqtt.connect();
  first.emit('connect');
  await connected;
  const pending = vi.fn();
  mqtt['publish']({ pattern: 'test', data: 'data' }, pending);

  first.emit('close');

  expect(pending).toHaveBeenCalledWith({
    err: expect.objectContaining({ message: 'Connection closed' }),
  });
  const reconnected = mqtt.connect();
  second.emit('connect');
  await reconnected;
  mqtt['publish']({ pattern: 'test', data: 'data' }, vi.fn());

  expect(second.listenerCount('message')).toBe(1);
  expect(second.subscribe).toHaveBeenCalledWith(
    'test/reply',
    expect.any(Function),
  );
});

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks @micalevisk, you were right, the reply state was still pointing at the dead client, i see kamil reset isInitialConnection and subscriptionsCount and failed the in flight callbacks in ab9b8f4 now, so the next client attaches the message listener and subscribes again

Comment on lines +210 to +215
// mqtt reconnects on the same client only while reconnectPeriod is set.
// Once it gives up, drop the client so the next connect() call starts
// over instead of returning the promise of a connection that is gone.
if (client === this.mqttClient && !client.reconnecting) {
this.discardClient(client);
}

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking. This also fires for the close that our own close() triggers through endAsync(), while close() is still awaiting it. On master a connect() made in that window returns the cached promise; on this branch the listener has already nulled mqttClient, so that connect() creates a second, live client, and when close() resumes it sets mqttClient = null again and that client is never ended (checked with fakes: createClient called twice, mqttClient null afterwards, endAsync never called on the second one).

mqtt.js sets disconnecting before it tears the stream down, and it is still true when close fires, so the smallest fix is to leave a close we asked for alone:

Suggested change
// mqtt reconnects on the same client only while reconnectPeriod is set.
// Once it gives up, drop the client so the next connect() call starts
// over instead of returning the promise of a connection that is gone.
if (client === this.mqttClient && !client.reconnecting) {
this.discardClient(client);
}
// mqtt reconnects on the same client only while reconnectPeriod is set.
// Once it gives up, drop the client so the next connect() call starts
// over instead of returning the promise of a connection that is gone.
// A close requested through end() is handled by close() itself.
if (
client === this.mqttClient &&
!client.reconnecting &&
!client.disconnecting
) {
this.discardClient(client);
}

(The alternative is close() detaching mqttClient before awaiting endAsync(), which would also fix the pre-existing stale-promise return in that window, but that is a bigger change than this PR needs.)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks @micalevisk, i see the window where our own close fires while close is still awaiting it, that guard was not part of the merge so i will keep the disconnecting check in mind if this path gets reworked

Comment on lines 109 to 112
this.pendingEventListeners.forEach(({ event, callback }) =>
this.mqttClient!.on(event, callback),
mqttClient.on(event, callback),
);
this.pendingEventListeners = [];

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Follow-up material, not for this PR unless you want it: listeners registered through ClientMqtt#on() are attached to the client of the first attempt and the list is emptied here, so after an automatic drop the replacement client has none of them, and the user has no signal that it happened (unlike after their own close()). #17828 solved the same thing for redis by keeping pendingEventListeners until close() and attaching them to every new pair.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good point @micalevisk, the redis pendingEventListeners shape looks like the right way to keep user listeners across an automatic drop, i will keep that in mind for a follow up

Comment on lines +543 to +560
it('should keep the client when mqtt reconnects on it', async () => {
const mqtt = new ClientMqtt({});
const untyped: any = mqtt;
const mqttClient = fakeClient(1000);
mqttClient.reconnecting = true;
const createClientStub = vi
.spyOn(mqtt, 'createClient')
.mockReturnValue(mqttClient);

const failed = mqtt.connect();
mqttClient.emit('error', refused());
await expect(failed).rejects.toThrow('connect ECONNREFUSED');

// mqtt keeps retrying on the same client, so it has to stay in place for
// the reconnect to be picked up.
expect(untyped.mqttClient).toBe(mqttClient);
expect(createClientStub).toHaveBeenCalledTimes(1);
});

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This case never emits close, so the !client.reconnecting guard in registerCloseListener is not exercised anywhere: I removed it and all 50 specs stayed green. Worth pinning, since that guard is what keeps a reconnectPeriod > 0 client alive across mqtt.js's reconnect loop (mqtt.js's own close handler sets reconnecting = true before ours runs, verified against the real driver):

it('should keep a closed client that is still reconnecting', () => {
  const mqtt = new ClientMqtt({});
  const untyped: any = mqtt;
  const mqttClient = fakeClient(1000);
  mqttClient.reconnecting = true;
  untyped.mqttClient = mqttClient;
  untyped.connectionPromise = Promise.resolve();

  mqtt.registerCloseListener(mqttClient);
  mqttClient.emit('close');

  expect(untyped.mqttClient).toBe(mqttClient);
  expect(untyped.connectionPromise).not.toBeNull();
});

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

agreed @micalevisk, the reconnecting guard should be pinned by a test, thanks for the concrete spec, i will add it if this code path comes back in a follow up

@kamilmysliwiec
kamilmysliwiec merged commit f80e0b6 into nestjs:master Sep 28, 2026
3 checks passed
@hktitof

hktitof commented Sep 28, 2026

Copy link
Copy Markdown
Contributor Author

thanks for merging this @kamilmysliwiec, and thanks @micalevisk for the review on the mqtt retry, you were right that the state still described the dead client, the reset in ab9b8f4 is exactly the shape i was missing

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants