Skip to content

[Bug] ConfigRpcTransportClient.shutdown() leaks nacos.client.config.listener.task threads by omitting multiTaskExecutor.shutdown()Β #15437

Closed

Description

@zmz789

Is this a Bug? Please describe.

In production we observed 1,663 nacos.client.config.listener.task-0 daemon threads accumulating over 3.5 hours on a nacos-client 2.5.2 application that repeatedly creates and shuts down ConfigService instances (one per monitoring task cycle).

Background

Each ConfigService.shutDown() call goes through:

NacosConfigService.worker.shutdown()
- ClientWorker.agent.shutdown() β†’ ConfigRpcTransportClient.shutdown()
- RpcClient gRPC connections β†’ closed
- executor.shutdown() β†’ closed (the while-loop scheduler)
- multiTaskExecutor β†’ never closed

multiTaskExecutor is a Map<String, ExecutorService> field in ConfigRpcTransportClient, lazily populated by ensureSyncExecutor(taskId):

// ClientWorker.java β€” ConfigRpcTransportClient
Map<String, ExecutorService> multiTaskExecutor = new HashMap<>();

private ExecutorService ensureSyncExecutor(String taskId) {
    if (!multiTaskExecutor.containsKey(taskId)) {
        multiTaskExecutor.put(taskId,
            new ThreadPoolExecutor(1, 1, 0L, TimeUnit.MILLISECONDS,
                new LinkedBlockingQueue<>(), r -> {
                    Thread t = new Thread(r, "nacos.client.config.listener.task-" + taskId);
                    t.setDaemon(true);
                    return t;
                }));
    }
    return multiTaskExecutor.get(taskId);
}

The executor is created on first checkListenCache() / checkRemoveListenCache() call (i.e., whenever the client has at least one listener registered). The shutdown() method closes executor but never touches multiTaskExecutor, so every ThreadPoolExecutor inside it keeps its daemon worker thread alive indefinitely β€” blocked on LinkedBlockingQueue.take(), waiting for tasks that will never come.

Why daemon threads do not self-exit

thread.setDaemon(true) only affects JVM shutdown behaviour. A daemon thread blocked inside a live ThreadPoolExecutor will not exit until the pool is explicitly shut down; it simply stays parked on the queue forever.

Reproduction

  1. Call NacosFactory.createConfigService(props).
  2. Register at least one listener via addListener() or getConfigAndSignListener() β€” this triggers ensureSyncExecutor("0") on the first executeConfigListen() pass.
  3. Call configService.shutDown().
  4. Repeat steps 1–3 in a loop.
  5. Observe nacos.client.config.listener.task-0 thread count in jstack growing monotonically.

Here is a validate file by Claude Code:

ClientWorkerShutdownRegressionTest.java

Affected versions

Confirmed on 2.5.2. The multiTaskExecutor field and ensureSyncExecutor method were introduced in the gRPC refactoring; all versions containing ConfigRpcTransportClient are likely affected.

Proposed fix

In ConfigRpcTransportClient.shutdown(), add cleanup of multiTaskExecutor alongside the existing executor.shutdown():

  // Before
  LOGGER.info("Shutdown executor {}", executor);
  executor.shutdown();

  // After
  LOGGER.info("Shutdown executor {}", executor);
  executor.shutdown();
  for (ExecutorService taskExec : multiTaskExecutor.values()) {
      taskExec.shutdownNow();
  }
  multiTaskExecutor.clear();

I am willing to submit a pull request for this fix.

Activity

  1. Zhengcy05 commented on Jun 27, 2026

    @Zhengcy05
    Contributor

    I noticed this issue has been fixed in version 3.x. This is a PR, I hope it helps.

  2. zmz789 commented on Jun 29, 2026

    @zmz789
    ContributorAuthor

    Thanks for @Zhengcy05's comment. Our infrastructure is partially still on Nacos 2.x server, so upgrading to the 3.x client isn't a best way for us right now.

    @KomachiSion Could you share whether there's any plan to backport the fix to the 2.x client SDK? A patch release for the 2.x line would be greatly appreciated.

  3. KomachiSion commented on Jun 29, 2026

    @KomachiSion
    Collaborator

    Thanks for @Zhengcy05's comment. Our infrastructure is partially still on Nacos 2.x server, so upgrading to the 3.x client isn't a best way for us right now.

    @KomachiSion Could you share whether there's any plan to backport the fix to the 2.x client SDK? A patch release for the 2.x line would be greatly appreciated.

    Welcome to cherry pick or fix in v2.x-develop

  4. zmz789 commented on Jun 29, 2026

    @zmz789
    ContributorAuthor

    Thanks, I've submitted a PR #15449 to backport this fix to v2.x-develop. The changes include two parts:

    1. Resource cleanup on shutdown (backported from [ISSUE #13546] help gcΒ #13646, credit to @FangYuan33)

    2. Concurrent safety of executor creation (backported from [Bug] Fix TOCTOU race condition and thread leak in ClientWorker.ensureSyncExecutor()Β #14927, credit to @LiyunZhang10)

  5. added a commit that references this issue on Jun 30, 2026
  6. added this to the 2.5.3 milestone on Jun 30, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions