Skip to content

[Bug]3.2.2版本集群模式下创建 SSE HTTP-to-MCP Server 时,持久化实例为不健康状态 #15477

Closed

Description

@xiaodong-chan

问题描述
在 Nacos 3.2.2集群的 MCP 管理页面中创建 MCP Server,配置如下:

  • 协议类型:SSE
  • 服务类型:HTTP 转 MCP 服务
  • 后端服务:选择“新建服务”
  • 后端地址:填写一个固定且可正常访问的地址

使用完全相同的后端地址和配置重复创建 MCP Server 时,Nacos 自动创建的 mcp-endpoints 持久化服务实例健康状态不稳定:

  • 有时创建后为“健康”
  • 有时创建后为“不健康”

该现象在后端地址、端口、协议及服务可用性均未变化的情况下出现。
根据当前实现,Direct Endpoint 创建的服务集群健康检查类型应为 NONE,新注册实例的默认健康状态应为 true。因此,实例健康状态不应依赖后端地址的 TCP/HTTP 可达性,也不应随机变成不健康。

Image

Activity

  1. changed the title [-][Bug]集群模式下创建 SSE HTTP-to-MCP Server 时,持久化实例为不健康状态[/-] [+][Bug]3.2.2版本集群模式下创建 SSE HTTP-to-MCP Server 时,持久化实例为不健康状态[/+] on Jul 8, 2026
  2. github-actions commented on Jul 15, 2026

    @github-actions

    This issue has been open 7 days with no activity. This will be closed in 7 days.

  3. Zhengcy05 commented on Jul 20, 2026

    @Zhengcy05
    Contributor

    From the current implementation, direct MCP endpoints are created with cluster health check type NONE, so a newly registered persistent instance should stay healthy by default and should not depend on backend TCP/HTTP reachability.

    The problem seems to be that the endpoint instance can still go through the normal persistent health-check path before the expected cluster metadata is effectively picked up. In that case, Naming falls back to the default cluster metadata, whose health-check type is TCP, and the instance may be marked unhealthy. After that, once the NONE metadata is in place, the none-health-check processor does not actively restore the instance health state, so the incorrect unhealthy status can remain.

    My proposed fix is to make sure MCP-owned direct endpoint services do not fall back to the default TCP health-check behavior during that window, and add a regression test for this path.

    If this direction looks good, I’d like to work on this and submit a PR.

  4. github-actions commented on Jul 28, 2026

    @github-actions

    This issue has been open 7 days with no activity. This will be closed in 7 days.

  5. KomachiSion commented on Aug 5, 2026

    @KomachiSion
    Collaborator

    From the current implementation, direct MCP endpoints are created with cluster health check type NONE, so a newly registered persistent instance should stay healthy by default and should not depend on backend TCP/HTTP reachability.

    The problem seems to be that the endpoint instance can still go through the normal persistent health-check path before the expected cluster metadata is effectively picked up. In that case, Naming falls back to the default cluster metadata, whose health-check type is TCP, and the instance may be marked unhealthy. After that, once the NONE metadata is in place, the none-health-check processor does not actively restore the instance health state, so the incorrect unhealthy status can remain.

    My proposed fix is to make sure MCP-owned direct endpoint services do not fall back to the default TCP health-check behavior during that window, and add a regression test for this path.

    If this direction looks good, I’d like to work on this and submit a PR.

    So is it means that before naming module load metadata snapshot, the healtch check will use default tcp check to do health check?

    But I'm confused, the instance and service is also stored by jraft, if no loading snapshot, how the service and instance appear?

  6. Zhengcy05 commented on Aug 10, 2026

    @Zhengcy05
    Contributor

    So is it means that before naming module load metadata snapshot, the healtch check will use default tcp check to do health check?

    Yes, in my view, it's default fallback

    Image Image

    But I'm confused, the instance and service is also stored by jraft, if no loading snapshot, how the service and instance appear?

    The service and instance can appear because MCP direct endpoint creation performs two separate writes: first serviceOperator.create(...) writes service metadata with the endpoint cluster set to NONE, then instanceOperator.registerInstance(...) writes the persistent instance. The possible window I meant is not “before snapshot loading”, but that the responsible node may apply/observe the persistent instance and schedule HealthCheckTaskV2 before the corresponding service metadata is visible in NamingMetadataManager.

    It's not that it didn't load, but rather that a gap appeared.

  7. added a commit that references this issue on Aug 10, 2026
    83e5aae
  8. KomachiSion commented on Aug 10, 2026

    @KomachiSion
    Collaborator

    But the mcp service will set the health check type as none. when register mcp service.

    see com.alibaba.nacos.ai.service.McpEndpointOperationService#doCreateNewService

  9. Zhengcy05 commented on Aug 10, 2026

    @Zhengcy05
    Contributor

    我的理解是核心在于时序问题,中间可能有Gap

    Image Image

    测试里“故意不调用 metadataManager.updateServiceMetadata(...)”模拟 health-check 执行这一刻,本节点本地 NamingMetadataManager 还没读到这份 metadata的情况。通过测试情况来看,这种场景是有可能的🤔

  10. KomachiSion commented on Aug 11, 2026

    @KomachiSion
    Collaborator

    但是我觉得修复方案通过修改默认值来进行处理是不是也不太合适? 再讨论下?

  11. Zhengcy05 commented on Aug 12, 2026

    @Zhengcy05
    Contributor

    但是我觉得修复方案通过修改默认值来进行处理是不是也不太合适? 再讨论下?

    是的,我也觉得修改默认值的行为不太合适,这样更改了语义,我的修改思路并不是修改默认值,提到的修改默认值的思路可能来源于PR #15680 ,如下图所示


    Image

    其实我的想法 #15681 更偏向于当 HealthCheckTaskV2 做 health check 时:

    metadataManager.getServiceMetadata(service)

    如果整个 ServiceMetadata 读不到,说明当前节点还没有可用的服务级 metadata 视图。这个时候不要构造默认 new ClusterMetadata(),而是跳过本轮 health check。

    Image

    不太确定我的思路是否在可以接受的范围🤔

  12. KomachiSion commented on Aug 13, 2026

    @KomachiSion
    Collaborator

    这样可能也不好, 因为ServiceMetadata本身就不是必须存在的吧。

  13. Zhengcy05 commented on Aug 13, 2026

    @Zhengcy05
    Contributor

    这样可能也不好, 因为ServiceMetadata本身就不是必须存在的吧。

    奥对,我刚刚去看了一下,确实没有相关语义明确 ServiceMetadata 必须存在,我还要再想一想其他方向🤔

  14. KomachiSion commented on Aug 17, 2026

    @KomachiSion
    Collaborator

    需要排查一下, 具体出现这个原因是什么?

    设置了None类型的健康检查,为什么重启的时候还会用默认的tcp去检查? 是不是健康检查需要添加一下前置, 如果Raft没有安装完snapshot 是不是应该快速退出等待下一轮?

  15. Zhengcy05 commented on Aug 18, 2026

    @Zhengcy05
    Contributor

    不好意思,最新在实习,手头上有一些别的事,有些问题推进比较慢❤️

    结合AI分析,应该是存在下面的启动竞态:

    • 持久化实例和服务 metadata 分别由 naming_persistent_service_v2、naming_service_metadata 两个独立的 Raft Group 恢复。
    • PersistentInstanceSnapshotOperation#loadSnapshot 在恢复 client 时会调用 snapshotClient.init(),此时就已经调度了 HealthCheckTaskV2,而 snapshot 加载方法尚未结束。
    • 当前健康检查链没有 Raft 或 snapshot readiness 前置。如果实例已经恢复,但服务 metadata 所属的 Raft Group 尚未完成恢复,getServiceMetadata() 会返回空,随后 HealthCheckTaskV2 构造默认的 ClusterMetadata,其健康检查类型为 TCP。
    • 如果这次 TCP 检查把实例标记为不健康,PersistentHealthStatusSynchronizer 会通过持久化实例 Raft Group 写回该状态。之后即使 NONE metadata 已经可见,NoneHealthCheckProcessor 也不会主动恢复实例的健康状态。

    用一个测试方法去模拟一下这个流程

    void testHealthCheckUsesDefaultTcpUntilServiceMetadataIsLoaded() {
            Service publishedService =
                Service.newService("public", "mcp-endpoint", "direct-service", false);
            String clusterName = "MCP_ENDPOINT";
            when(ipPortBasedClient.getAllPublishedService())
                .thenReturn(Collections.singletonList(publishedService));
            when(switchDomain.isHealthCheckEnabled(publishedService.getGroupedServiceName()))
                .thenReturn(true);
            when(ipPortBasedClient.getInstancePublishInfo(publishedService))
                .thenReturn(instancePublishInfo);
            when(instancePublishInfo.getCluster()).thenReturn(clusterName);
            healthCheckTaskV2.setCancelled(true);
            
            healthCheckTaskV2.doHealthCheck();
            
            ArgumentCaptor<ClusterMetadata> metadataCaptor =
                ArgumentCaptor.forClass(ClusterMetadata.class);
            verify(processorDelegate).process(eq(healthCheckTaskV2), eq(publishedService),
                metadataCaptor.capture());
            assertEquals(Tcp.TYPE, metadataCaptor.getValue().getHealthyCheckType());
            
            ClusterMetadata noneClusterMetadata = new ClusterMetadata();
            noneClusterMetadata.setHealthyCheckType(AbstractHealthChecker.None.TYPE);
            noneClusterMetadata.setHealthChecker(new AbstractHealthChecker.None());
            ServiceMetadata serviceMetadata = new ServiceMetadata();
            serviceMetadata.getClusters().put(clusterName, noneClusterMetadata);
            metadataManager.updateServiceMetadata(publishedService, serviceMetadata);
            clearInvocations(processorDelegate);
            
            healthCheckTaskV2.doHealthCheck();
            
            verify(processorDelegate).process(eq(healthCheckTaskV2), eq(publishedService),
                metadataCaptor.capture());
            assertEquals(AbstractHealthChecker.None.TYPE,
                metadataCaptor.getValue().getHealthyCheckType());
        }

    因此,我认同健康检查在相关 Raft 状态尚未恢复完成时应该快速退出,等待下一轮执行。
    当前的 CPProtocol#isReady() 不能直接作为这个前置条件:JRaftServer#isReady() 目前只检查 Raft Server 是否启动,在 strict mode 下额外检查 leader 是否存在,并不能表示 naming_service_metadata Group 已经完成 snapshot 安装。
    我现在考虑新增 interceptor🤔

  16. KomachiSion commented on Aug 24, 2026

    @KomachiSion
    Collaborator

    interceptor应该已经保留了拓展的能力,应该实现起来是比较容易的, 欢迎按照这个方案提交PR。

  17. nacos-community commented on Aug 25, 2026

    @nacos-community
    Collaborator

    Issue Evaluation

    Category: kind/research | Status: Needs Info

    The report identifies a plausible defect: Direct Endpoint–backed mcp-endpoints persistent instances should use health check type NONE and default to healthy, yet the instance health flips non-deterministically in cluster mode even with an unchanged, reachable backend.

    To verify the cluster-mode registration/health-check wiring, please provide:

    • Server logs (naming + MCP module) around the moment the instance turns unhealthy
    • Whether the outcome correlates with which cluster node handled the create request
    • Instance detail (health checker type, enabled, lastRefTime) when it shows unhealthy
    • Whether the backend address is reachable from all cluster nodes

    With this information we can cross-check the MCP direct-endpoint registration path in cluster mode.


    Automated evaluation by github-manager-bot

  18. added a commit that references this issue on Aug 31, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions