Skip to content

Why does the Nacos client write to the disk cache synchronously after receiving NotifySubscriberRequest? #15321

Closed

Description

@MajorHe1
Image

这个设计应该很多年了,我理解99.99%的情况都是从内存中读取服务信息,这里是不是应该优化成异步写入磁盘缓存?考虑到磁盘IO慢,高频大规模推送的时候还卡,服务端迟迟收不到ACK,经常会有 ConnectionBusyException抛出来。

Image

Activity

  1. added
    kind/questionCategory issues related to questions or problems
    area/ClientRelated to Nacos Client SDK
    on Jun 8, 2026
  2. Zhengcy05 commented on Jun 8, 2026

    @Zhengcy05
    Contributor

    Hi👋, I think your point makes sense, and I agree there is room for optimization here. That said, I still have a few concerns. If we change this path to async persistence, it may also introduce some follow-up problems that need to be thought through first.

    The main three points I can think of are:

    1. Consistency semantics

    The current behavior is very straightforward: when the ACK is returned, both the in-memory state and the local disk cache have already been updated.

    If this is changed to async, the meaning of the ACK would become “the client has received the push and finished updating memory”, while the disk cache may still not be flushed yet. If the process exits during that window, the local snapshot may still be stale.

    This may still be acceptable, because runtime reads are mainly served from memory anyway, and the disk cache is more like a best-effort snapshot for restart preload / failover. But I think we should first make it clear whether this semantic change is acceptable.

    Possible options might be:

    • explicitly treat the disk cache as an eventually consistent snapshot
    • perform a bounded flush during shutdown
    • or provide a sync / async switch for compatibility with different expectations
    1. Ordering / overwrite problem

    The same service may receive multiple pushes in a short period of time.

    If the implementation simply submits every update to a thread pool for async disk write, we may end up with an older snapshot being written after a newer one.

    So I feel this may be better modeled as coalescing by serviceKey: only keep the latest snapshot pending in memory, and let a background worker flush them in order. That way, at least the final persisted state is the newest one, instead of writing every intermediate state to disk.

    1. Backpressure problem

    If every high-frequency push creates an independent async task, then even though the main thread is no longer blocked, the task queue itself may build up. In that case, the problem is only shifted from synchronous blocking to asynchronous backlog.

    So if we want to make this async, it probably should be more than a simple executor.submit(writeTask). We may need to consider:

    • deduplicating pending serviceKeys
    • background batch flushing
    • a bounded queue, or at least a bounded state set
    • allowing intermediate states to be dropped under overload, while still preserving the latest state

    I think this will be a significant change, and we may need more data to support the claim that it will indeed create a performance bottleneck.

  3. KomachiSion commented on Jun 8, 2026

    @KomachiSion
    Collaborator

    It's a great sugeestion and idea.

    I think we can use event to async refresh and write disk cache to replace sync update cache.

  4. Zhengcy05 commented on Jun 8, 2026

    @Zhengcy05
    Contributor

    I want to try🙋, please assign it to me.

  5. MajorHe1 commented on Jun 8, 2026

    @MajorHe1
    CollaboratorAuthor

    I want to try🙋, please assign it to me.

    It has already been assigned to you, looking forward to your excellent contribution.

  6. Zhengcy05 commented on Jun 8, 2026

    @Zhengcy05
    Contributor

    It's a great sugeestion and idea.

    I think we can use event to async refresh and write disk cache to replace sync update cache.

    Just to make sure I understand the direction correctly: when you mentioned using event to async refresh and write disk cache, do you mean reusing the existing event mechanism (for example NotifyCenter / subscriber), or more generally an event-driven async persistence flow inside the naming client?

    I want to align the implementation direction first.

  7. added a commit that references this issue on Jun 16, 2026
  8. added
    kind/enhancementCategory issues or prs related to enhancement.
    and removed
    kind/questionCategory issues related to questions or problems
    on Jun 16, 2026
  9. added this to the 3.2.3 milestone on Jun 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

area/ClientRelated to Nacos Client SDKkind/enhancementCategory issues or prs related to enhancement.

Type

No type

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions