Repository navigation
Description
Is this a Bug? Please describe.
In production we observed 1,663 nacos.client.config.listener.task-0 daemon threads accumulating over 3.5 hours on a nacos-client 2.5.2 application that repeatedly creates and shuts down ConfigService instances (one per monitoring task cycle).
Background
Each ConfigService.shutDown() call goes through:
NacosConfigService.worker.shutdown()
- ClientWorker.agent.shutdown() β ConfigRpcTransportClient.shutdown()
- RpcClient gRPC connections β closed
- executor.shutdown() β closed (the while-loop scheduler)
- multiTaskExecutor β never closed
multiTaskExecutor is a Map<String, ExecutorService> field in ConfigRpcTransportClient, lazily populated by ensureSyncExecutor(taskId):
// ClientWorker.java β ConfigRpcTransportClient
Map<String, ExecutorService> multiTaskExecutor = new HashMap<>();
private ExecutorService ensureSyncExecutor(String taskId) {
if (!multiTaskExecutor.containsKey(taskId)) {
multiTaskExecutor.put(taskId,
new ThreadPoolExecutor(1, 1, 0L, TimeUnit.MILLISECONDS,
new LinkedBlockingQueue<>(), r -> {
Thread t = new Thread(r, "nacos.client.config.listener.task-" + taskId);
t.setDaemon(true);
return t;
}));
}
return multiTaskExecutor.get(taskId);
}The executor is created on first checkListenCache() / checkRemoveListenCache() call (i.e., whenever the client has at least one listener registered). The shutdown() method closes executor but never touches multiTaskExecutor, so every ThreadPoolExecutor inside it keeps its daemon worker thread alive indefinitely β blocked on LinkedBlockingQueue.take(), waiting for tasks that will never come.
Why daemon threads do not self-exit
thread.setDaemon(true) only affects JVM shutdown behaviour. A daemon thread blocked inside a live ThreadPoolExecutor will not exit until the pool is explicitly shut down; it simply stays parked on the queue forever.
Reproduction
- Call NacosFactory.createConfigService(props).
- Register at least one listener via addListener() or getConfigAndSignListener() β this triggers ensureSyncExecutor("0") on the first executeConfigListen() pass.
- Call configService.shutDown().
- Repeat steps 1β3 in a loop.
- Observe nacos.client.config.listener.task-0 thread count in jstack growing monotonically.
Here is a validate file by Claude Code:
ClientWorkerShutdownRegressionTest.java
Affected versions
Confirmed on 2.5.2. The multiTaskExecutor field and ensureSyncExecutor method were introduced in the gRPC refactoring; all versions containing ConfigRpcTransportClient are likely affected.
Proposed fix
In ConfigRpcTransportClient.shutdown(), add cleanup of multiTaskExecutor alongside the existing executor.shutdown():
// Before
LOGGER.info("Shutdown executor {}", executor);
executor.shutdown();
// After
LOGGER.info("Shutdown executor {}", executor);
executor.shutdown();
for (ExecutorService taskExec : multiTaskExecutor.values()) {
taskExec.shutdownNow();
}
multiTaskExecutor.clear();I am willing to submit a pull request for this fix.
Activity
Zhengcy05 commentedon Jun 27, 2026
I noticed this issue has been fixed in version 3.x. This is a PR, I hope it helps.
zmz789 commentedon Jun 29, 2026
Thanks for @Zhengcy05's comment. Our infrastructure is partially still on Nacos 2.x server, so upgrading to the 3.x client isn't a best way for us right now.
@KomachiSion Could you share whether there's any plan to backport the fix to the 2.x client SDK? A patch release for the 2.x line would be greatly appreciated.
KomachiSion commentedon Jun 29, 2026
Welcome to cherry pick or fix in v2.x-develop
zmz789 commentedon Jun 29, 2026
Thanks, I've submitted a PR #15449 to backport this fix to v2.x-develop. The changes include two parts:
Resource cleanup on shutdown (backported from [ISSUE #13546] help gcΒ #13646, credit to @FangYuan33)
Concurrent safety of executor creation (backported from [Bug] Fix TOCTOU race condition and thread leak in ClientWorker.ensureSyncExecutor()Β #14927, credit to @LiyunZhang10)
[ISSUE #15437] fix: backport thread leak and executor safety fixes toβ¦