Repository navigation
[ISSUE #15477] Defer persistent health checks until service metadata is ready - #15784
Conversation
|
Thanks for your this PR. 🙏 感谢您提交的PR。 🙏 |
KomachiSion
left a comment
There was a problem hiding this comment.
Thanks for the update. I traced the startup and snapshot recovery paths again.
The current readiness check based on LEADER_META_DATA is not sufficient. It only shows that a leader is known for the Raft group; it does not mean that this node has completed loading the local snapshot or that the local naming state is ready.
Could you please adjust the implementation as follows?
-
Keep the current
HealthCheckTaskV2type filter. In the production path, ephemeral clients useClientBeatCheckTaskV2, while persistent clients useHealthCheckTaskV2, so an additional client-type check is unnecessary. -
Add a monotonic
snapshotLoadedstate for both related CP processors:PersistentClientOperationServiceImpl/naming_persistent_service_v2ServiceMetadataProcessor/naming_service_metadata
The state should initially be
falseand becometrueonly after the correspondingonSnapshotLoad()completes successfully. Once it becomestrue, it should not be reset during the process lifetime. -
Please expose the state from the two Spring-managed processors rather than keeping it only in a temporary
SnapshotOperationinstance.loadSnapshotOperate()is called more than once and currently creates a new operation each time, so storing readiness only in one returned operation can cause the interceptor to observe a different instance. -
The interceptor should quickly skip the health check while either snapshot is not loaded. Please log every skipped task and include at least:
- task id
- persistent client snapshot state
- service metadata snapshot state
- application startup state
-
To avoid permanently disabling health checks when no local snapshot exists, use
ApplicationUtils.isStarted()as a fail-open condition. Once both snapshots are loaded, or the application has completed startup, set a monotonichealthCheckReleasedflag and never close the gate again.Since
ApplicationUtils.startedis written by the startup thread and read by health-check threads, please make itvolatileor replace it with anAtomicBoolean.
Please also add tests covering:
- either snapshot is not loaded before application startup: intercepted;
- both snapshots are loaded: passed;
- no snapshot is loaded but application startup has completed: passed;
- once released, the gate never closes again;
- snapshot load failure does not mark the snapshot as loaded;
- ephemeral heartbeat tasks remain unaffected.
This change addresses the local startup snapshot-ordering race described by this PR. ApplicationUtils.isStarted() should be documented as a fail-open fallback, not as a general guarantee that both Raft groups have caught up to their leaders after startup.
KomachiSion
left a comment
There was a problem hiding this comment.
These is only one problem for PR.
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
|
By the way, Does the codecov must be 100%, or is it sufficient to reach a certain threshold. |
Best to improve the coverage to 100% |
What is the purpose of the change
This PR fixes a startup race that can incorrectly mark persistent MCP endpoint instances as unhealthy.
Persistent instances and service metadata are recovered by two independent Raft groups. During persistent client snapshot recovery,
HealthCheckTaskV2may start before the localnaming_service_metadatagroup has completed recovery. Missing metadata is then interpreted as the default TCP health checker, and a failed check may persist an incorrect unhealthy state.This change adds an interceptor that defers persistent instance health checks until the service metadata Raft group is ready. It only applies to
HealthCheckTaskV2and preserves the existing fallback behavior after metadata recovery completes.Fixes #15477.
Brief changelog
ServiceMetadataReadyInterceptorto gate persistent health checks on service metadata readiness.Verifying this change
The following checks passed:
mvn -pl naming spotless:applymvn -pl naming spotless:checkmvn -pl naming -Dtest=ServiceMetadataReadyInterceptorTest,HealthCheckTaskInterceptWrapperTest,HealthCheckTaskV2Test testmvn -B clean compile apache-rat:check checkstyle:check spotbugs:check spotless:check -DskipTestsFollow this checklist to help us incorporate your contribution quickly and easily:
[ISSUE #123] Fix UnknownException when host config not exist. Each commit in the pull request should have a meaningful subject line and body.mvn spotless:applyto format your code. Runmvn -B clean compile apache-rat:check checkstyle:check spotbugs:check spotless:check -DskipTeststo make sure basic checks pass. Runmvn clean installto make sure unit-test pass. Runmvn clean test-compile failsafe:integration-testto make sure integration-test pass.