Skip to content

[Bug] Plugin availability API returns only self=true and all remote nodes=false in Nacos 3.2.4 clusterΒ #15836

Closed

Description

@xiaowang872

Bug Report

Describe the bug

In a 3-node Nacos 3.2.4 cluster, the new Console Plugin Management page
reports all enabled plugins as available on only 1 / 3 nodes.

This is not limited to the MySQL datasource plugin. Multiple enabled plugins
show the same result, for example:

datasource-dialect:mysql    1 / 3
ai-storage:nacos_config     1 / 3
auth:nacos                  1 / 3
control:nacos               1 / 3
visibility:nacos            1 / 3
ai-pipeline:skill-scanner   1 / 3

The three Nacos nodes themselves are running normally.

More importantly, querying /v3/console/plugin/availability directly against
each individual Console node shows that each node reports only itself as
available, while the other two nodes are reported as unavailable.

Environment

  • Nacos Server: 3.2.4
  • Docker image: nacos/nacos-server:v3.2.4
  • Java: 17
  • Deployment mode: cluster
  • Kubernetes: Tencent Cloud TKE
  • Replicas: 3
  • Deployment type: StatefulSet
  • External datasource: MySQL
  • Host mode: hostname

Cluster members:

nacos-engine-0.nacos-hs.engine.svc.cluster.local:8848
nacos-engine-1.nacos-hs.engine.svc.cluster.local:8848
nacos-engine-2.nacos-hs.engine.svc.cluster.local:8848

The three pods use the same datasource configuration:

SPRING_DATASOURCE_PLATFORM=mysql
MYSQL_SERVICE_HOST=<same MySQL host>
MYSQL_SERVICE_PORT=<same MySQL port>
MYSQL_SERVICE_DB_NAME=nacos_engine
MYSQL_SERVICE_USER=<same user>
PREFER_HOST_MODE=hostname

The Nacos server identity key/value are also configured consistently across
all three cluster members.

What has already been verified

All three Nacos pods are Running and stable.

Configuration persistence works normally. A test configuration created from
the Nacos Console:

Data ID: test-nacos.yaml
Group: DEFAULT_GROUP

is successfully persisted into the external MySQL config_info table.

Inter-node TCP connectivity has also been verified successfully between the
Nacos pods:

8848  OK
9848  OK
9849  OK

After the cluster became stable, nacos.log did not show continuing errors
matching:

Server check fail
Fail to connect server
UNAVAILABLE
grpc fail
cluster fail

Steps to reproduce

  1. Deploy a 3-node Nacos 3.2.4 cluster.
  2. Use hostname mode with a Kubernetes Headless Service.
  3. Configure all three nodes to use the same MySQL datasource.
  4. Wait until all three nodes are Running and the cluster is stable.
  5. Log in to the Nacos Console.
  6. Open Plugin Management.
  7. Observe that enabled plugins show 1 / 3 available nodes.
  8. Query the plugin availability API directly on each Nacos Console node.

Example:

curl -sS \
  -H "Authorization: Bearer $TOKEN" \
  'http://nacos-engine-0.nacos-hs.engine.svc.cluster.local:8080/v3/console/plugin/availability?pluginType=datasource-dialect&pluginName=mysql'

Result from nacos-engine-0:

{
  "code": 0,
  "message": "success",
  "data": {
    "nacos-engine-2.nacos-hs.engine.svc.cluster.local:8848": false,
    "nacos-engine-1.nacos-hs.engine.svc.cluster.local:8848": false,
    "nacos-engine-0.nacos-hs.engine.svc.cluster.local:8848": true
  }
}

Querying nacos-engine-1:

{
  "code": 0,
  "message": "success",
  "data": {
    "nacos-engine-2.nacos-hs.engine.svc.cluster.local:8848": false,
    "nacos-engine-1.nacos-hs.engine.svc.cluster.local:8848": true,
    "nacos-engine-0.nacos-hs.engine.svc.cluster.local:8848": false
  }
}

Querying nacos-engine-2:

{
  "code": 0,
  "message": "success",
  "data": {
    "nacos-engine-2.nacos-hs.engine.svc.cluster.local:8848": true,
    "nacos-engine-1.nacos-hs.engine.svc.cluster.local:8848": false,
    "nacos-engine-0.nacos-hs.engine.svc.cluster.local:8848": false
  }
}

Therefore the behavior is deterministic:

request node-0 -> node-0=true, node-1=false, node-2=false
request node-1 -> node-0=false, node-1=true, node-2=false
request node-2 -> node-0=false, node-1=false, node-2=true

Each node appears to recognize its local plugin correctly, but remote-node
plugin availability is always reported as false.

Expected behavior

According to the Nacos Console API documentation,
GET /v3/console/plugin/availability returns:

Map<node address, availability>

When the same enabled plugin is available on all three healthy Nacos nodes,
the expected result should be similar to:

{
  "nacos-engine-0.nacos-hs.engine.svc.cluster.local:8848": true,
  "nacos-engine-1.nacos-hs.engine.svc.cluster.local:8848": true,
  "nacos-engine-2.nacos-hs.engine.svc.cluster.local:8848": true
}

and the Console should display:

3 / 3

Actual behavior

Each Console node reports only its own plugin as available:

local node  = true
remote node = false
remote node = false

The Console therefore always displays:

1 / 3

for all enabled plugins.

Additional observations

This does not currently appear to affect basic Nacos Config functionality.

  • All three Nacos server pods remain healthy.
  • MySQL persistence works.
  • Configurations created through the Console are written successfully to
    MySQL.
  • Inter-node 8848/9848/9849 TCP connectivity is working.

Because the same symptom occurs for multiple unrelated plugins, this appears
more likely to be related to the cluster-level plugin availability query
mechanism rather than the MySQL datasource plugin itself.

The current Nacos internal RPC specification documents:

PluginAvailabilityRequest
PluginAvailabilityResponse

as the internal RPC used to query plugin availability on a node.

Could you please confirm whether there is a known issue in Nacos 3.2.4 with
cross-node PluginAvailability queries, RPC dispatch, payload registration,
or plugin availability aggregation?

Official documentation references

Console Plugin Availability API:

https://nacos.io/docs/latest/manual/admin/console-api/

Nacos internal RPC specification:

https://github.com/alibaba/nacos/blob/develop/specs/en/design/foundation-internal-rpc-spec.md

Nacos 3.2.4 release:

https://github.com/alibaba/nacos/releases/tag/3.2.4

Image

Activity

  1. KomachiSion commented on Sep 10, 2026

    @KomachiSion
    Collaborator

    It should be your network has problem, your nacos-server cluster nodes can't connect to each other.

  2. xiaowang872 commented on Sep 11, 2026

    @xiaowang872
    Author

    Thanks for the reply.

    I checked the cluster networking again.

    The Nacos cluster nodes can discover each other through the headless service, and peer-finder returns all 3 nodes correctly:

    nacos-engine-0.nacos-svc.engine.svc.cluster.local
    nacos-engine-1.nacos-svc.engine.svc.cluster.local
    nacos-engine-2.nacos-svc.engine.svc.cluster.local

    Nacos also starts with the complete server list:

    The server IP list of Nacos is [
    nacos-engine-0.nacos-svc.engine.svc.cluster.local:8848,
    nacos-engine-1.nacos-svc.engine.svc.cluster.local:8848,
    nacos-engine-2.nacos-svc.engine.svc.cluster.local:8848
    ]

    So basic DNS/service discovery between the three nodes is working.

    However, the plugin availability API still reports only the local node as available, while the other two nodes are false.

    Could you please clarify which port/protocol or internal API is used by the plugin availability check between Nacos nodes?

    For example, should I specifically verify connectivity on:

    • 8848
    • 9848
    • 9849
    • 7848
    • 8080

    If there is a specific endpoint or log keyword for the remote plugin availability probe, I can test that directly and provide the result.
    If this is still considered a network issue, could you point me to the exact inter-node request used by the plugin availability API? I can capture the request/response or test it from each pod.
    Environment:

    • Nacos Server: 3.2.4
    • Kubernetes StatefulSet: 3 replicas
    • Cluster mode
    • Headless Service
    • preferHostnameOverIp=true
  3. KomachiSion commented on Sep 14, 2026

    @KomachiSion
    Collaborator

    should we see the logs and find the some inforamtion?

  4. github-actions commented on Sep 21, 2026

    @github-actions

    This issue has been open 7 days with no activity. This will be closed in 7 days.

  5. github-actions commented on Oct 1, 2026

    @github-actions

    This issue has been open 7 days with no activity. This will be closed in 7 days.

  6. added 2 commits that reference this issue on Oct 8, 2026
  7. added this to the 3.3.0 milestone on Oct 8, 2026
  8. added a commit that references this issue on Oct 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area/pluginkind/bugCategory issues or prs related to bug.

    Type

    No type

    Projects

    No projects

      Milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions