Skip to content

feat: add WithMessageFilter option to filter messages early in the notification pipeline - #678

Merged
MarcoPolo merged 12 commits into
libp2p:masterfrom
ideo-org:feat/filter-self-messages
Mar 18, 2026
Merged

MarcoPolo merged 12 commits into
libp2p:masterfrom
ideo-org:feat/filter-self-messages

Conversation

@laciferin2024

Copy link
Copy Markdown
Contributor

Summary

Adds a WithSelfNotification(enabled bool) subscribe option that allows subscribers to opt out of receiving messages published by the local node. This addresses a common use case where applications need to avoid processing their own published messages.

Closes #537

Changes

  • Added WithSelfNotification(enabled bool) as a new SubOpt for Topic.Subscribe()
  • When set to false, messages published by the local host are filtered out in notifySubs before delivery
  • Default behavior is unchanged β€” subscriptions still receive all messages, including self-published ones
  • Added test TestWithSelfNotification covering:
    • Self-published messages are skipped on filtered subscriptions
    • Self-published messages still arrive on default subscriptions
    • Messages from remote peers still arrive on filtered subscriptions

Design

The filtering happens at the notifySubs level rather than in Next(), so self-published messages never enter the subscription channel buffer. This avoids wasting buffer space and prevents unnecessary drops when the subscriber is slow.

Each subscription stores the local host's peer ID at creation time and compares it against msg.ReceivedFrom during delivery.

…hed messages

Add a SubOpt that allows subscribers to opt out of receiving messages
published by the local node. This is useful for applications where
processing self-published messages is unnecessary overhead.

The default behavior is unchanged - subscriptions still receive all
messages including self-published ones. Passing WithSelfNotification(false)
filters them out at the delivery level in notifySubs.

Closes libp2p#537
Comment thread pubsub.go Resolved
@MarcoPolo

Copy link
Copy Markdown
Contributor

Did you need this feature? Why was skipping the message on the application side not good enough?

@laciferin2024

Copy link
Copy Markdown
Contributor Author

Did you need this feature? Why was skipping the message on the application side not good enough?

The issue with filtering on the application side (i.e. in the handler after Next()) is that the message has already been delivered into the subscription's channel buffer. For slow subscribers this means the buffer fills up with self-published messages that the app will just discard, which can cause legitimate remote messages to be dropped. Filtering at notifySubs prevents self-published messages from ever entering the buffer, so the buffer capacity is fully available for messages the subscriber actually wants. That was the main reason to do it at this layer rather than leaving it to the caller.

@MarcoPolo

Copy link
Copy Markdown
Contributor

Realistically a single equality check by the handler is not going to slow the subscriber enough to cause it to drop messages. Did you see this issue in production?

@laciferin2024
laciferin2024 force-pushed the feat/filter-self-messages branch from 798fd35 to ee12f38 Compare March 3, 2026 05:41
@laciferin2024

Copy link
Copy Markdown
Contributor Author

Realistically a single equality check by the handler is not going to slow the subscriber enough to cause it to drop messages. Did you see this issue in production?

With tight buffer sizes yes, this filter auto drops self messages.

Just to give u my story:

  1. Processing in heart beat intervals with 'N' goroutines . A large fraction of those goroutines are consumed by self messages that are dropped.

@MarcoPolo

MarcoPolo commented Mar 3, 2026 •

Copy link
Copy Markdown
Contributor

Let's change this to be an option that sets a message subscription filter function that can be passed in as a SubOpt. Then the application can filter out messages where ReceivedFrom == self, and this can happen in notifySubs

@vyzo

vyzo commented Mar 3, 2026

Copy link
Copy Markdown
Collaborator

precedent from IP Multicast here too, this is a sockopt in POSIX.

@twenty-tools
twenty-tools force-pushed the feat/filter-self-messages branch from 464d6f7 to c3b94c8 Compare March 3, 2026 22:41
@laciferin2024

Copy link
Copy Markdown
Contributor Author

Let's change this to be an option that sets a message subscription filter function that can be passed in as a SubOpt. Then the application can filter out messages where ReceivedFrom == self, and this can happen in notifySubs

Done

Comment thread pubsub.go Resolved
@twenty-tools
twenty-tools force-pushed the feat/filter-self-messages branch from cbe0d71 to f7d6c24 Compare March 17, 2026 09:51
@twenty-tools
twenty-tools force-pushed the feat/filter-self-messages branch from f7d6c24 to 96545f5 Compare March 17, 2026 09:57
@MarcoPolo MarcoPolo changed the title feat: add WithSelfNotification option to filter self-published messages feat: add WithMessageFilter option to filter messages early in the notification pipeline Mar 18, 2026
@MarcoPolo
MarcoPolo merged commit a2ec129 into libp2p:master Mar 18, 2026
7 checks passed
@laciferin2024

Copy link
Copy Markdown
Contributor Author

Hurray πŸŽ‰, thank you @MarcoPolo for the patience and snappy review.

MarcoPolo added a commit that referenced this pull request Apr 17, 2026
This release introduces support for the partial messages extension. The
partial messages extension gives the application precise control on
exchanging message metadata and exchanging parts of messages. Unlike
normal Gossipsub messages, Partial Messages are not opaque and peers can
exchange message metadata separately from message data. An example use
case is reconcilling when peers have some or all of a message. This
works in a backwards compatible way.

Notable changes:

- Message filter option β€” WithMessageFilter to filter messages early in the notification pipeline (#678).
- Fanout-only topics β€” New FanoutOnly topic option (#676).
- Dynamic direct peer handling β€” Threadsafe dynamic direct peer management for GossipSub (#673).

Fixes & improvements:

- Structured RPC logging
- Fix leaked state with OnNewIncomingStream/OnClosedIncomingStream (with failing test first)
- Renamed {Add/Remove}Peer β†’ On{New/Closed}OutboundStream
- partialmessages: init fanout if empty (#690)
- Log errors when handling RPC in extensions (#668)
- Ensure Hello Packet is first RPC sent; set write deadline for outgoing messages
- Added Rpc.From()
MarcoPolo added a commit that referenced this pull request Apr 17, 2026
This release introduces support for the partial messages extension. The
partial messages extension gives the application precise control on
exchanging message metadata and exchanging parts of messages. Unlike
normal Gossipsub messages, Partial Messages are not opaque and peers can
exchange message metadata separately from message data. An example use
case is reconcilling when peers have some or all of a message. This
works in a backwards compatible way.

Notable changes:

- New FanoutOnly topic option (#676).
- WithMessageFilter to filter messages early in the notification pipeline (#678).

Fixes & improvements:

- Structured RPC logging
- Fix leaked state with OnNewIncomingStream/OnClosedIncomingStream (with failing test first)
- Renamed {Add/Remove}Peer β†’ On{New/Closed}OutboundStream
- partialmessages: init fanout if empty (#690)
- Log errors when handling RPC in extensions (#668)
- Ensure Hello Packet is first RPC sent; set write deadline for outgoing messages
- Added Rpc.From()
- Threadsafe dynamic direct peer management for GossipSub (#673).
@MarcoPolo MarcoPolo mentioned this pull request Apr 17, 2026
MarcoPolo added a commit that referenced this pull request Apr 18, 2026
This release introduces support for the partial messages extension. The
partial messages extension gives the application precise control on
exchanging message metadata and exchanging parts of messages. Unlike
normal Gossipsub messages, Partial Messages are not opaque and peers can
exchange message metadata separately from message data. An example use
case is reconcilling when peers have some or all of a message. This
works in a backwards compatible way.

Notable changes:

- New FanoutOnly topic option (#676).
- WithMessageFilter to filter messages early in the notification pipeline (#678).

Fixes & improvements:

- Structured RPC logging
- Fix leaked state with OnNewIncomingStream/OnClosedIncomingStream (with failing test first)
- Renamed {Add/Remove}Peer β†’ On{New/Closed}OutboundStream
- partialmessages: init fanout if empty (#690)
- Log errors when handling RPC in extensions (#668)
- Ensure Hello Packet is first RPC sent; set write deadline for outgoing messages
- Added Rpc.From()
- Threadsafe dynamic direct peer management for GossipSub (#673).
MarcoPolo added a commit that referenced this pull request Apr 20, 2026
This release introduces support for the partial messages extension. The
partial messages extension gives the application precise control on
exchanging message metadata and exchanging parts of messages. Unlike
normal Gossipsub messages, Partial Messages are not opaque and peers can
exchange message metadata separately from message data. An example use
case is reconcilling when peers have some or all of a message. This
works in a backwards compatible way.

Notable changes:

- New FanoutOnly topic option (#676).
- WithMessageFilter to filter messages early in the notification pipeline (#678).

Fixes & improvements:

- Structured RPC logging
- Fix leaked state with OnNewIncomingStream/OnClosedIncomingStream (with failing test first)
- Renamed {Add/Remove}Peer β†’ On{New/Closed}OutboundStream
- partialmessages: init fanout if empty (#690)
- Log errors when handling RPC in extensions (#668)
- Ensure Hello Packet is first RPC sent; set write deadline for outgoing messages
- Added Rpc.From()
- Threadsafe dynamic direct peer management for GossipSub (#673).
MarcoPolo added a commit that referenced this pull request Apr 20, 2026
This release introduces support for the partial messages extension. The
partial messages extension gives the application precise control on
exchanging message metadata and exchanging parts of messages. Unlike
normal Gossipsub messages, Partial Messages are not opaque and peers can
exchange message metadata separately from message data. An example use
case is reconcilling when peers have some or all of a message. This
works in a backwards compatible way.

Notable changes:

- New FanoutOnly topic option (#676).
- WithMessageFilter to filter messages early in the notification pipeline (#678).

Fixes & improvements:

- Structured RPC logging
- Fix leaked state with OnNewIncomingStream/OnClosedIncomingStream (with failing test first)
- Renamed {Add/Remove}Peer β†’ On{New/Closed}OutboundStream
- partialmessages: init fanout if empty (#690)
- Log errors when handling RPC in extensions (#668)
- Ensure Hello Packet is first RPC sent; set write deadline for outgoing messages
- Added Rpc.From()
- Threadsafe dynamic direct peer management for GossipSub (#673).
MarcoPolo added a commit that referenced this pull request Apr 23, 2026
This release introduces support for the partial messages extension. The
partial messages extension gives the application precise control on
exchanging message metadata and exchanging parts of messages. Unlike
normal Gossipsub messages, Partial Messages are not opaque and peers can
exchange message metadata separately from message data. An example use
case is reconcilling when peers have some or all of a message. This
works in a backwards compatible way.

Notable changes:

- New FanoutOnly topic option (#676).
- WithMessageFilter to filter messages early in the notification
pipeline (#678).

Fixes & improvements:

- Structured RPC logging
- Fix leaked state with OnNewIncomingStream/OnClosedIncomingStream (with
failing test first)
- Renamed {Add/Remove}Peer β†’ On{New/Closed}OutboundStream
- partialmessages: init fanout if empty (#690)
- Log errors when handling RPC in extensions (#668)
- Ensure Hello Packet is first RPC sent; set write deadline for outgoing
messages
- Added Rpc.From()
- Threadsafe dynamic direct peer management for GossipSub (#673).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Next function returns own messages

3 participants