Repository navigation
Closed
Labels
P1High: Likely tackled by core team if no one steps upHigh: Likely tackled by core team if no one steps updif/expertExtensive knowledge (implications, ramifications) requiredExtensive knowledge (implications, ramifications) requiredeffort/weeksEstimated to take multiple weeksEstimated to take multiple weeksstatus/in-progressIn progressIn progress
Description
Per ipfs/kubo#10837, we are sending new blocks to the reprovider regardless of strategies.
The places were this happens are:
- providing.Exchange.NotifyNewBlocks:
- Calls the underlying exchange's NotifyNewBlocks. This is bitswap:
- The server is informed that new blocks are available to provide over Bitswap
- The client is informed that new blocks have arrived, therefore no need to keep waiting for them
- Calls
Provide()on the provider.
- Calls the underlying exchange's NotifyNewBlocks. This is bitswap:
- exchange.NotifyNewBlocks is called from:
- BlockService:
- AddBlock(s)
- GetBlock(s)
- Bitswap doesn't write blocks themselves, all is mediated from Blockservice.
- BlockService:
My proposal is to distribute the responsability of providing new blocks to the places that play a role in the different strategies:
- Providing Blockstore:
- Relevant for 'all' or
flatstrategies, should know when new blocks are written:- Unlike the BlockService, that provides blocks on Get/Put, because it can write a block in the process of Getting it, the BlockStore can just provide on writes.
- Providing Pinner:
- Relevant for the
pinned,rootsstrategy:- When pinning something, provides every block as dag is traversed, or provides only the root at the end, depending on the strategy.
- Relevant for the
- Providing MFS:
- Relevant for the
mfsstrategy. - I have to think about it. We could provide everytime we do dagService.Add(), when strategy enabled.
- Relevant for the
- Relevant for 'all' or
The process would be as follows:
- Add options to Blockstore, Pinner, MFS. Example:
- WithRootsProvider(p Provider)
- WithPinnedProvider(p Provider)
- Call the Provide operation on the necessary places when the option has been set.
- Remove NotifyNewBlocks from BlockService and Remove providingExchange.
- Fix tests,
- On Kubo, pass the provider to the Pinner/MFS/Blockstore during setup, depending on what strategy is set, or leave it disabled otherwise.
The result is that Pinner/MFS/Blockstore will call provide when configured to do so.
Reacted by Daniel Norman and Andrew Gillis
Reacted by Guillaume Michel
Metadata
Metadata
Assignees
Labels
P1High: Likely tackled by core team if no one steps upHigh: Likely tackled by core team if no one steps updif/expertExtensive knowledge (implications, ramifications) requiredExtensive knowledge (implications, ramifications) requiredeffort/weeksEstimated to take multiple weeksEstimated to take multiple weeksstatus/in-progressIn progressIn progress
Activity
guillaumemichel commentedon Jun 25, 2025
The plan seems great! I fully support this approach to decentralizing the provide logic. Distributing responsibility to the relevant components (Blockstore, Pinner, and MFS) according to each strategy will make the system cleaner and get rid of the bug (providing every received block).
IMO it makes sense to use a Provider interface that is compatible with both the old
boxo/providerinterface as well as the newgo-libp2p-kad-dhtreprovider sweep, see libp2p/go-libp2p-kad-dht#1093.[-]Decentralizing new blocks notification[/-][+]Decentralizing new blocks notification (avoid providing everything)[/+]guillaumemichel commentedon Jul 9, 2025
It may be relevant to handle together these two other locations where
Provide()is called in kuboRemove providing Exchange. Call Provide() from relevant places.
2color commentedon Aug 7, 2025
How does #976 affect this issue?
hsanjuan commentedon Aug 8, 2025
It closes it :)