Repository navigation
fix(microservices): let the mqtt client retry after a failed connect - #17891
kamilmysliwiec merged 2 commits into
Conversation
when reconnectPeriod is 0, mqtt gives up after the first failed connection and stays closed on the same client, so the rejected attempt stays cached and every later connect() call returns that same rejection instead of trying again, the same as the redis, rmq, kafka and nats clients got in the sibling fixes drop the client when a failed attempt cannot reconnect on its own and when a closed client gave up reconnecting, so the next connect() call starts over with a new client
micalevisk
left a comment
There was a problem hiding this comment.
Thanks for closing the gap the kafka fix (#17816) left open, and for the probe in the description: it reproduces with mqtt@5.16.0 exactly as described, and the two new "drop" specs fail on master and pass here.
I ran this against a real checkout: npx vitest run packages/microservices (63 files, 932 tests green, the 2 new specs red once the base client-mqtt.ts is swapped in), npm run build, npm run lint:ci, prettier on both files. The integration/ MQTT suites need the mosquitto container from docker-compose.yml and were not run. I also drove the real driver against a closed port to check the event order this relies on: with reconnectPeriod: 0, error fires (with reconnecting === false) one loop turn before close, so the .catch branch discards first and the close listener finds the client already gone; with reconnectPeriod > 0, mqtt.js's own close handler runs _setupReconnect() before ours, so reconnecting is already true when registerCloseListener looks at it and the client is kept. The guards hold.
One blocking issue, though. The close-listener path is new behaviour (a connection that was established and then dropped with reconnectPeriod: 0 now gets replaced on the next connect()), but discardClient() only resets mqttClient and connectionPromise. isInitialConnection stays true, so the replacement client never gets the message listener; subscriptionsCount keeps the counts of the dead connection, so publish() skips subscribe() on the new client; and the requests still in routingMap are never failed. Every send() after the "recovery" hangs while the status reads CONNECTED. The redis fix in this same series (#17828, discardClients()) resets wasInitialConnectionSuccessful for exactly this reason, and close() here already does the same reset. Details, a repro and a spec in the inline comment.
Non-blocking, but please take a look while you are in there:
connect()issued whileclose()awaitsendAsync()now creates a second live client thatclose()then forgets (inline comment on the close listener).- The
!client.reconnectingguard is not pinned by any spec: removing it leaves all 50 tests green (inline comment on the spec). - Listeners registered through
ClientMqtt#on()stay on the dropped client; #17828 kept them for later clients. Fine as a follow-up (inline comment).
No docs follow-up needed. "No breaking change" is fair; the only visible change besides the retry is that unwrap() throws Not initialized after an automatic drop, same as ClientNats does since #17861.
| private discardClient(client: MqttClient) { | ||
| if (client !== this.mqttClient) { | ||
| return; | ||
| } | ||
| this.mqttClient = null; | ||
| this.connectionPromise = null; | ||
| } |
There was a problem hiding this comment.
Blocking. For the first-connect failure this is enough, but this method is also what the new close listener calls after a connection that was established and then dropped with reconnectPeriod: 0. In that case the next connect() does create a new client, as intended, but the rest of the state still describes the dead one:
isInitialConnectionstaystrue, soregisterConnectListenernever attaches themessagelistener to the replacement client (client-mqtt.ts:227-230). Replies are never routed.subscriptionsCountkeeps the counts of the dropped connection, sopublish()takes thesubscriptionsCount > 0branch and never callssubscribe()on the new client (client-mqtt.ts:354-360). The broker never delivers the replies to it either.- The callbacks still in
routingMapare never failed, unlike inclose().
Every send() after the "recovery" hangs, with the status at CONNECTED. Reproduced with fakes on this branch: after connect → one in-flight request → close → connect, the second client has listenerCount('message') === 0, subscribe is not called for the second request, and a reply emitted on the second client reaches no callback.
close() (lines 71-75) already resets exactly this, and the redis twin of this fix (#17828, discardClients()) resets wasInitialConnectionSuccessful for the same reason. On #17691 the argument for resetting isInitialConnection only in close() was that an ended client never emits connect again; here a new client will.
| private discardClient(client: MqttClient) { | |
| if (client !== this.mqttClient) { | |
| return; | |
| } | |
| this.mqttClient = null; | |
| this.connectionPromise = null; | |
| } | |
| private discardClient(client: MqttClient) { | |
| if (client !== this.mqttClient) { | |
| return; | |
| } | |
| this.mqttClient = null; | |
| this.connectionPromise = null; | |
| // Same reset as close(): the next client has to get the response | |
| // listener again and subscribe to the reply channels anew, and the | |
| // requests still waiting on the dropped connection cannot be answered. | |
| this.isInitialConnection = false; | |
| this.subscriptionsCount.clear(); | |
| this.handleClose(); | |
| } |
And a spec for it (fails on this branch, passes with the reset), next to the other three:
it('should serve requests again after a dropped connection is replaced', async () => {
const mqtt = new ClientMqtt({});
const untyped: any = mqtt;
const first = fakeClient(0);
const second = fakeClient(0);
for (const c of [first, second]) {
c.subscribe = vi.fn((_channel, done) => done());
c.publish = vi.fn();
}
vi.spyOn(mqtt, 'createClient')
.mockReturnValueOnce(first)
.mockReturnValueOnce(second);
vi.spyOn(mqtt, 'assignPacketId' as any).mockImplementation(packet =>
Object.assign(packet as object, { id: '1' }),
);
const connected = mqtt.connect();
first.emit('connect');
await connected;
const pending = vi.fn();
mqtt['publish']({ pattern: 'test', data: 'data' }, pending);
first.emit('close');
expect(pending).toHaveBeenCalledWith({
err: expect.objectContaining({ message: 'Connection closed' }),
});
const reconnected = mqtt.connect();
second.emit('connect');
await reconnected;
mqtt['publish']({ pattern: 'test', data: 'data' }, vi.fn());
expect(second.listenerCount('message')).toBe(1);
expect(second.subscribe).toHaveBeenCalledWith(
'test/reply',
expect.any(Function),
);
});There was a problem hiding this comment.
thanks @micalevisk, you were right, the reply state was still pointing at the dead client, i see kamil reset isInitialConnection and subscriptionsCount and failed the in flight callbacks in ab9b8f4 now, so the next client attaches the message listener and subscribes again
| // mqtt reconnects on the same client only while reconnectPeriod is set. | ||
| // Once it gives up, drop the client so the next connect() call starts | ||
| // over instead of returning the promise of a connection that is gone. | ||
| if (client === this.mqttClient && !client.reconnecting) { | ||
| this.discardClient(client); | ||
| } |
There was a problem hiding this comment.
Non-blocking. This also fires for the close that our own close() triggers through endAsync(), while close() is still awaiting it. On master a connect() made in that window returns the cached promise; on this branch the listener has already nulled mqttClient, so that connect() creates a second, live client, and when close() resumes it sets mqttClient = null again and that client is never ended (checked with fakes: createClient called twice, mqttClient null afterwards, endAsync never called on the second one).
mqtt.js sets disconnecting before it tears the stream down, and it is still true when close fires, so the smallest fix is to leave a close we asked for alone:
| // mqtt reconnects on the same client only while reconnectPeriod is set. | |
| // Once it gives up, drop the client so the next connect() call starts | |
| // over instead of returning the promise of a connection that is gone. | |
| if (client === this.mqttClient && !client.reconnecting) { | |
| this.discardClient(client); | |
| } | |
| // mqtt reconnects on the same client only while reconnectPeriod is set. | |
| // Once it gives up, drop the client so the next connect() call starts | |
| // over instead of returning the promise of a connection that is gone. | |
| // A close requested through end() is handled by close() itself. | |
| if ( | |
| client === this.mqttClient && | |
| !client.reconnecting && | |
| !client.disconnecting | |
| ) { | |
| this.discardClient(client); | |
| } |
(The alternative is close() detaching mqttClient before awaiting endAsync(), which would also fix the pre-existing stale-promise return in that window, but that is a bigger change than this PR needs.)
There was a problem hiding this comment.
thanks @micalevisk, i see the window where our own close fires while close is still awaiting it, that guard was not part of the merge so i will keep the disconnecting check in mind if this path gets reworked
| this.pendingEventListeners.forEach(({ event, callback }) => | ||
| this.mqttClient!.on(event, callback), | ||
| mqttClient.on(event, callback), | ||
| ); | ||
| this.pendingEventListeners = []; |
There was a problem hiding this comment.
Follow-up material, not for this PR unless you want it: listeners registered through ClientMqtt#on() are attached to the client of the first attempt and the list is emptied here, so after an automatic drop the replacement client has none of them, and the user has no signal that it happened (unlike after their own close()). #17828 solved the same thing for redis by keeping pendingEventListeners until close() and attaching them to every new pair.
There was a problem hiding this comment.
good point @micalevisk, the redis pendingEventListeners shape looks like the right way to keep user listeners across an automatic drop, i will keep that in mind for a follow up
| it('should keep the client when mqtt reconnects on it', async () => { | ||
| const mqtt = new ClientMqtt({}); | ||
| const untyped: any = mqtt; | ||
| const mqttClient = fakeClient(1000); | ||
| mqttClient.reconnecting = true; | ||
| const createClientStub = vi | ||
| .spyOn(mqtt, 'createClient') | ||
| .mockReturnValue(mqttClient); | ||
|
|
||
| const failed = mqtt.connect(); | ||
| mqttClient.emit('error', refused()); | ||
| await expect(failed).rejects.toThrow('connect ECONNREFUSED'); | ||
|
|
||
| // mqtt keeps retrying on the same client, so it has to stay in place for | ||
| // the reconnect to be picked up. | ||
| expect(untyped.mqttClient).toBe(mqttClient); | ||
| expect(createClientStub).toHaveBeenCalledTimes(1); | ||
| }); |
There was a problem hiding this comment.
This case never emits close, so the !client.reconnecting guard in registerCloseListener is not exercised anywhere: I removed it and all 50 specs stayed green. Worth pinning, since that guard is what keeps a reconnectPeriod > 0 client alive across mqtt.js's reconnect loop (mqtt.js's own close handler sets reconnecting = true before ours runs, verified against the real driver):
it('should keep a closed client that is still reconnecting', () => {
const mqtt = new ClientMqtt({});
const untyped: any = mqtt;
const mqttClient = fakeClient(1000);
mqttClient.reconnecting = true;
untyped.mqttClient = mqttClient;
untyped.connectionPromise = Promise.resolve();
mqtt.registerCloseListener(mqttClient);
mqttClient.emit('close');
expect(untyped.mqttClient).toBe(mqttClient);
expect(untyped.connectionPromise).not.toBeNull();
});There was a problem hiding this comment.
agreed @micalevisk, the reconnecting guard should be pinned by a test, thanks for the concrete spec, i will add it if this code path comes back in a follow up
|
thanks for merging this @kamilmysliwiec, and thanks @micalevisk for the review on the mqtt retry, you were right that the state still described the dead client, the reset in ab9b8f4 is exactly the shape i was missing |
PR Checklist
Please check if your PR fulfills the following requirements:
PR Type
What kind of change does this PR introduce?
What is the current behavior?
when reconnectPeriod is 0, mqtt does not retry after the first failed connection, it emits error and close and stays closed on the same client. connect() caches the rejected attempt in connectionPromise and keeps mqttClient, so every later connect() call sees mqttClient set and returns that same rejection instead of trying again. the only way out is a close() first
this reproduces with the real mqtt driver on master against a dead port. the first connect rejects, then the second connect creates no new client and rejects with the same cached error
Issue Number: N/A
Steps to reproduce
What is the new behavior?
a failed attempt only drops the client when mqtt will not reconnect on its own, and a closed client that gave up reconnecting is dropped too, so the next connect() starts over with a new client. when reconnectPeriod is set, mqtt keeps retrying on the same client and nothing is dropped, so the background reconnect still resolves the promise on connect. the same probe now creates a new client on the second call
Does this PR introduce a breaking change?
Other information
the redis, rmq, kafka and nats clients all reset a failed attempt already, mqtt was the one client left caching it