Skip to content

fix(microservices): let the redis client retry after a failed connect - #17828

Merged
kamilmysliwiec merged 4 commits into
nestjs:masterfrom
hktitof:fix/redis-connect-retry-exhausted
Sep 22, 2026
Merged

kamilmysliwiec merged 4 commits into
nestjs:masterfrom
hktitof:fix/redis-connect-retry-exhausted

Conversation

@hktitof

@hktitof hktitof commented Sep 21, 2026

Copy link
Copy Markdown
Contributor

Context: sibling to #17816 (which let the kafka client retry after a failed connect). This applies the same rule to the redis client when the retries are exhausted

PR Checklist

Please check if your PR fulfills the following requirements:

PR Type

What kind of change does this PR introduce?

  • Bugfix
  • Feature
  • Code style update (formatting, local variables)
  • Refactoring (no functional changes, no api changes)
  • Build related changes
  • CI related changes
  • Other... Please describe:

What is the current behavior?

ClientRedis#connect caches the connection promise and never clears it when the attempt fails:

this.connectionPromise = this.handleConnection();

when retryAttempts is configured and the retries are exhausted, the end listener replaces the cached promise with a rejected Error: Connection lost. Trying to reconnect... promise, and every later connect() call replays that rejection instead of trying again, even once the server is reachable. Only close() clears it. The pub or sub client the failed attempt managed to create is also left behind without being closed

probed on untouched master with a real ioredis client against a closed port (127.0.0.1:59999, retryAttempts: 1):

PROBE: err1=Connection is closed. | after1=CACHED raw:Error: Connection lost. Trying to reconnect... | err2=raw:Error: Connection lost. Trying to reconnect... | createClientCalls=2

the second connect() call made no new attempt (2 createClient calls in total) and rejected with the cached string

Issue Number: N/A

Steps to reproduce

  1. npx vitest run packages/microservices/test/client/client-redis.spec.ts on master, the three tests in connect after a failed attempt (the last one uses a real ioredis client against a closed port, the other two stub it)
  2. Expected: a failed attempt leaves no cached promise, tears down the clients it created, and the next connect() tries again
  3. Actual (raw output on untouched master):
 FAIL  packages/microservices/test/client/client-redis.spec.ts > ClientRedis > connect after a failed attempt > should discard the partial connection when an attempt fails
AssertionError: expected Promise{…} to be null

- Expected:
null

+ Received:
Promise {}

 FAIL  packages/microservices/test/client/client-redis.spec.ts > ClientRedis > connect after a failed attempt > should try again on the next call instead of caching a failed attempt
Error: connection refused

 FAIL  packages/microservices/test/client/client-redis.spec.ts > ClientRedis > connect after a failed attempt > should try again on the next call after the retries are exhausted
AssertionError: expected 'Error: Connection lost. Trying to reconnect…' to be an instance of Error

 Test Files  1 failed (1)
      Tests  3 failed | 42 passed (45)

What is the new behavior?

A rejected attempt resets connectionPromise to null and quits the pub/sub clients the attempt managed to create (Promise.allSettled, same shape as the kafka fix), so the next connect() starts over. The end and reconnect listeners keep the existing reconnect model after a drop untouched, the catch only runs when the attempt itself fails

Does this PR introduce a breaking change?

  • Yes
  • No

Other information

the three new tests fail on master (3 failed | 42 passed) and pass with the fix (45 passed), every microservices test file is green with the fix: 898 tests across 63 files. nats resets its promise in a catch and tcp resets when the socket closes, so this brings redis in line with them

hktitof and others added 4 commits September 21, 2026 15:40
When retryAttempts is configured and the retries are exhausted, the failed
attempt stays cached in connectionPromise, so every later connect() call
replays the cached "connection lost" rejection instead of trying again, even
once the server is reachable. The clients the failed attempt managed to
create are also left behind without being closed.

Reset the cached promise when the attempt fails and quit the clients it
created, so the next call starts over, same as the kafka client got in nestjs#17816
Resetting the cached promise in a catch around the attempt missed how
the redis client manages its state: the "reconnecting", "ready" and
"end" listeners rewrite connectionPromise and the pub/sub clients.

- When ioredis gives up on a client ("end", retryAttempts not set or
  exhausted), drop the pair, reset connectionPromise and
  wasInitialConnectionSuccessful and disconnect the other client, so
  the next connect() call starts over. With retryAttempts set, a
  "Connection lost" rejection stayed cached forever, and without it
  the new sub client never got the response listener.
- Only reset in the catch when no listener took over connectionPromise,
  so ioredis keeps retrying a failed first attempt in the background
  instead of the clients being quit right away.
- Ignore the events of clients that are no longer the current pair, so
  a dropped or closed client shutting down late cannot re-cache a
  rejection, drop the new clients, or throw on a null sub client.
- Keep the listeners registered with on() and attach them to every
  pair.

Replace the no-op stubs in the tests with fakes that emit the
connection events, and the test that relied on a closed port with one
running a real ioredis client against a local server that drops every
connection.
ioredis leaves a client that is disconnected while waiting to reconnect in the "reconnecting" status, so wait for the connection attempts to stop instead of for every client to reach "end".
@kamilmysliwiec
kamilmysliwiec merged commit e715176 into nestjs:master Sep 22, 2026
4 checks passed
kamilmysliwiec added a commit that referenced this pull request Sep 24, 2026
…17865)

* fix(microservices): let the rmq client retry after a failed connect

When the first connection attempt fails, the rejection stays cached in connectionPromise,
so every later connect() call replays the error of the first attempt until close() is
called, even once the broker is reachable again. The connection manager the attempt
created also stays behind and keeps retrying in the background

Reset the cached promise when the attempt fails and close the manager it created, so the
next call starts over, same as the kafka client got in #17816 and the redis client in
#17828

* fix(microservices): keep a newer rmq connection when a stale one fails

---------

Co-authored-by: Kamil Myśliwiec <mail@kamilmysliwiec.com>
kamilmysliwiec pushed a commit to nestjs/docs.nestjs.com that referenced this pull request Sep 29, 2026
…3559)

Listeners registered with the client's on() method stay registered until
close() and are attached to each new socket (TCP, v12.1.1, nestjs/nest#17888)
or pub/sub pair (Redis, v12.1, nestjs/nest#17828). Registrations are not
deduplicated, so re-registering from within a callback accumulates them.
kamilmysliwiec pushed a commit that referenced this pull request Sep 29, 2026
the mqtt client drops the listeners registered through on() when the client
that gave up reconnecting gets replaced. the next connect() only reattaches
the listeners it kept from before the first connect, so a listener added
after that never sees the events of the new client. this is the same defect
#17888 fixed for the tcp client and #17828 fixed for the redis client

keep the on() registrations until close() and reattach them to every client
created later, including the listeners added after the initial connection
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants