Skip to content

[ISSUE #15477] Defer persistent health checks until service metadata is ready - #15784

Merged
KomachiSion merged 6 commits into
alibaba:developfrom
Zhengcy05:fix/mcp-endpoint-health-check
Aug 31, 2026
Merged

KomachiSion merged 6 commits into
alibaba:developfrom
Zhengcy05:fix/mcp-endpoint-health-check

Conversation

@Zhengcy05

Copy link
Copy Markdown
Contributor

What is the purpose of the change

This PR fixes a startup race that can incorrectly mark persistent MCP endpoint instances as unhealthy.

Persistent instances and service metadata are recovered by two independent Raft groups. During persistent client snapshot recovery, HealthCheckTaskV2 may start before the local naming_service_metadata group has completed recovery. Missing metadata is then interpreted as the default TCP health checker, and a failed check may persist an incorrect unhealthy state.

This change adds an interceptor that defers persistent instance health checks until the service metadata Raft group is ready. It only applies to HealthCheckTaskV2 and preserves the existing fallback behavior after metadata recovery completes.

Fixes #15477.

Brief changelog

  • Add ServiceMetadataReadyInterceptor to gate persistent health checks on service metadata readiness.
  • Register the interceptor through the existing health-check SPI.
  • Add unit tests for unavailable, ready, and failed readiness checks.
  • Keep ephemeral heartbeat health checks unaffected.
  • Document the recovery readiness requirement in the English and Chinese naming specifications.

Verifying this change

The following checks passed:

  • mvn -pl naming spotless:apply
  • mvn -pl naming spotless:check
  • mvn -pl naming -Dtest=ServiceMetadataReadyInterceptorTest,HealthCheckTaskInterceptWrapperTest,HealthCheckTaskV2Test test
    • 21 tests passed.
  • mvn -B clean compile apache-rat:check checkstyle:check spotbugs:check spotless:check -DskipTests
    • All 61 modules passed.

Follow this checklist to help us incorporate your contribution quickly and easily:

  • Make sure there is a Github issue filed for the change (usually before you start working on it). Trivial changes like typos do not require a Github issue. Your pull request should address just this issue, without pulling in other changes - one PR resolves one issue.
  • Format the pull request title like [ISSUE #123] Fix UnknownException when host config not exist. Each commit in the pull request should have a meaningful subject line and body.
  • Write a pull request description that is detailed enough to understand what the pull request does, how, and why.
  • Write necessary unit-test to verify your logic correction, more mock a little better when cross module dependency exist. If the new feature or significant change is committed, please remember to add integration-test in [test module](https://github.com/alibaba/nacos/tree/master/test).
  • Run mvn spotless:apply to format your code. Run mvn -B clean compile apache-rat:check checkstyle:check spotbugs:check spotless:check -DskipTests to make sure basic checks pass. Run mvn clean install to make sure unit-test pass. Run mvn clean test-compile failsafe:integration-test to make sure integration-test pass.

@github-actions

Copy link
Copy Markdown

Thanks for your this PR. 🙏
Please check again for your PR changes whether contains any usage/api/configuration change such as Add new API , Add new configuration, Change default value of configuration.
If so, please add or update documents(markdown type) in docs/next/ for repository nacos-group/nacos-group.github.io


感谢您提交的PR。 🙏
请再次查看您的PR内容,确认是否包含任何使用方式/API/配置参数的变更,如:新增API、新增配置参数、修改默认配置等操作。
如果是,请确保在提交之前,在仓库nacos-group/nacos-group.github.io中的docs/next/目录下添加或更新文档(markdown格式)。

@KomachiSion KomachiSion left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the update. I traced the startup and snapshot recovery paths again.

The current readiness check based on LEADER_META_DATA is not sufficient. It only shows that a leader is known for the Raft group; it does not mean that this node has completed loading the local snapshot or that the local naming state is ready.

Could you please adjust the implementation as follows?

  1. Keep the current HealthCheckTaskV2 type filter. In the production path, ephemeral clients use ClientBeatCheckTaskV2, while persistent clients use HealthCheckTaskV2, so an additional client-type check is unnecessary.

  2. Add a monotonic snapshotLoaded state for both related CP processors:

    • PersistentClientOperationServiceImpl / naming_persistent_service_v2
    • ServiceMetadataProcessor / naming_service_metadata

    The state should initially be false and become true only after the corresponding onSnapshotLoad() completes successfully. Once it becomes true, it should not be reset during the process lifetime.

  3. Please expose the state from the two Spring-managed processors rather than keeping it only in a temporary SnapshotOperation instance. loadSnapshotOperate() is called more than once and currently creates a new operation each time, so storing readiness only in one returned operation can cause the interceptor to observe a different instance.

  4. The interceptor should quickly skip the health check while either snapshot is not loaded. Please log every skipped task and include at least:

    • task id
    • persistent client snapshot state
    • service metadata snapshot state
    • application startup state
  5. To avoid permanently disabling health checks when no local snapshot exists, use ApplicationUtils.isStarted() as a fail-open condition. Once both snapshots are loaded, or the application has completed startup, set a monotonic healthCheckReleased flag and never close the gate again.

    Since ApplicationUtils.started is written by the startup thread and read by health-check threads, please make it volatile or replace it with an AtomicBoolean.

Please also add tests covering:

  • either snapshot is not loaded before application startup: intercepted;
  • both snapshots are loaded: passed;
  • no snapshot is loaded but application startup has completed: passed;
  • once released, the gate never closes again;
  • snapshot load failure does not mark the snapshot as loaded;
  • ephemeral heartbeat tasks remain unaffected.

This change addresses the local startup snapshot-ordering race described by this PR. ApplicationUtils.isStarted() should be documented as a fail-open fallback, not as a general guarantee that both Raft groups have caught up to their leaders after startup.

@KomachiSion KomachiSion left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These is only one problem for PR.

@codecov-commenter

codecov-commenter commented Aug 31, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 84.78261% with 7 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
...k/interceptor/ServiceMetadataReadyInterceptor.java 79.41% 3 Missing and 4 partials ⚠️

📢 Thoughts on this report? Let us know!

@Zhengcy05

Copy link
Copy Markdown
Contributor Author

By the way, Does the codecov must be 100%, or is it sufficient to reach a certain threshold.

@KomachiSion
KomachiSion merged commit 95c2ddf into alibaba:develop Aug 31, 2026
7 checks passed
@KomachiSion

Copy link
Copy Markdown
Collaborator

By the way, Does the codecov must be 100%, or is it sufficient to reach a certain threshold.

Best to improve the coverage to 100%

@KomachiSion KomachiSion added kind/bug Category issues or prs related to bug. area/MCP area/Naming labels Aug 31, 2026
@KomachiSion KomachiSion added this to the 3.3.0 milestone Aug 31, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/MCP area/Naming kind/bug Category issues or prs related to bug.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]3.2.2版本集群模式下创建 SSE HTTP-to-MCP Server 时,持久化实例为不健康状态

3 participants