Skip to content

Decentralizing new blocks notification (avoid providing everything)Β #963

Closed

Description

@hsanjuan

Per ipfs/kubo#10837, we are sending new blocks to the reprovider regardless of strategies.

The places were this happens are:

  • providing.Exchange.NotifyNewBlocks:
    • Calls the underlying exchange's NotifyNewBlocks. This is bitswap:
      • The server is informed that new blocks are available to provide over Bitswap
      • The client is informed that new blocks have arrived, therefore no need to keep waiting for them
    • Calls Provide() on the provider.
  • exchange.NotifyNewBlocks is called from:
    • BlockService:
      • AddBlock(s)
      • GetBlock(s)
    • Bitswap doesn't write blocks themselves, all is mediated from Blockservice.

My proposal is to distribute the responsability of providing new blocks to the places that play a role in the different strategies:

  • Providing Blockstore:
    • Relevant for 'all' or flat strategies, should know when new blocks are written:
      • Unlike the BlockService, that provides blocks on Get/Put, because it can write a block in the process of Getting it, the BlockStore can just provide on writes.
    • Providing Pinner:
      • Relevant for the pinned, roots strategy:
        • When pinning something, provides every block as dag is traversed, or provides only the root at the end, depending on the strategy.
    • Providing MFS:
      • Relevant for the mfs strategy.
      • I have to think about it. We could provide everytime we do dagService.Add(), when strategy enabled.

The process would be as follows:

  • Add options to Blockstore, Pinner, MFS. Example:
    • WithRootsProvider(p Provider)
    • WithPinnedProvider(p Provider)
  • Call the Provide operation on the necessary places when the option has been set.
  • Remove NotifyNewBlocks from BlockService and Remove providingExchange.
    • Fix tests,
  • On Kubo, pass the provider to the Pinner/MFS/Blockstore during setup, depending on what strategy is set, or leave it disabled otherwise.

The result is that Pinner/MFS/Blockstore will call provide when configured to do so.

Activity

  1. guillaumemichel commented on Jun 25, 2025

    @guillaumemichel
    Member

    The plan seems great! I fully support this approach to decentralizing the provide logic. Distributing responsibility to the relevant components (Blockstore, Pinner, and MFS) according to each strategy will make the system cleaner and get rid of the bug (providing every received block).

    • Add options to Blockstore, Pinner, MFS. Example:
      • WithRootsProvider(p Provider)
      • WithPinnedProvider(p Provider)

    IMO it makes sense to use a Provider interface that is compatible with both the old boxo/provider interface as well as the new go-libp2p-kad-dht reprovider sweep, see libp2p/go-libp2p-kad-dht#1093.

  2. changed the title [-]Decentralizing new blocks notification[/-] [+]Decentralizing new blocks notification (avoid providing everything)[/+] on Jun 29, 2025
  3. guillaumemichel commented on Jul 9, 2025

    @guillaumemichel
    Member
  4. added
    P1High: Likely tackled by core team if no one steps up
    dif/expertExtensive knowledge (implications, ramifications) required
    effort/weeksEstimated to take multiple weeks
    and removed
    need/triageNeeds initial labeling and prioritization
    on Jul 29, 2025
  5. 2color commented on Aug 7, 2025

    @2color
    Member

    How does #976 affect this issue?

  6. hsanjuan commented on Aug 8, 2025

    @hsanjuan
    ContributorAuthor

    How does #976 affect this issue?

    It closes it :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

P1High: Likely tackled by core team if no one steps updif/expertExtensive knowledge (implications, ramifications) requiredeffort/weeksEstimated to take multiple weeksstatus/in-progressIn progress

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions