Repository navigation
feat: add WithMessageFilter option to filter messages early in the notification pipeline - #678
Conversation
β¦hed messages Add a SubOpt that allows subscribers to opt out of receiving messages published by the local node. This is useful for applications where processing self-published messages is unnecessary overhead. The default behavior is unchanged - subscriptions still receive all messages including self-published ones. Passing WithSelfNotification(false) filters them out at the delivery level in notifySubs. Closes libp2p#537
|
Did you need this feature? Why was skipping the message on the application side not good enough? |
The issue with filtering on the application side (i.e. in the handler after Next()) is that the message has already been delivered into the subscription's channel buffer. For slow subscribers this means the buffer fills up with self-published messages that the app will just discard, which can cause legitimate remote messages to be dropped. Filtering at notifySubs prevents self-published messages from ever entering the buffer, so the buffer capacity is fully available for messages the subscriber actually wants. That was the main reason to do it at this layer rather than leaving it to the caller. |
|
Realistically a single equality check by the handler is not going to slow the subscriber enough to cause it to drop messages. Did you see this issue in production? |
798fd35 to
ee12f38
Compare
β¦tification" This reverts commit ee12f38.
With tight buffer sizes yes, this filter auto drops self messages. Just to give u my story:
|
|
Let's change this to be an option that sets a message subscription filter function that can be passed in as a SubOpt. Then the application can filter out messages where ReceivedFrom == self, and this can happen in notifySubs |
|
precedent from IP Multicast here too, this is a sockopt in POSIX. |
464d6f7 to
c3b94c8
Compare
Done |
cbe0d71 to
f7d6c24
Compare
f7d6c24 to
96545f5
Compare
|
Hurray π, thank you @MarcoPolo for the patience and snappy review. |
This release introduces support for the partial messages extension. The partial messages extension gives the application precise control on exchanging message metadata and exchanging parts of messages. Unlike normal Gossipsub messages, Partial Messages are not opaque and peers can exchange message metadata separately from message data. An example use case is reconcilling when peers have some or all of a message. This works in a backwards compatible way. Notable changes: - Message filter option β WithMessageFilter to filter messages early in the notification pipeline (#678). - Fanout-only topics β New FanoutOnly topic option (#676). - Dynamic direct peer handling β Threadsafe dynamic direct peer management for GossipSub (#673). Fixes & improvements: - Structured RPC logging - Fix leaked state with OnNewIncomingStream/OnClosedIncomingStream (with failing test first) - Renamed {Add/Remove}Peer β On{New/Closed}OutboundStream - partialmessages: init fanout if empty (#690) - Log errors when handling RPC in extensions (#668) - Ensure Hello Packet is first RPC sent; set write deadline for outgoing messages - Added Rpc.From()
This release introduces support for the partial messages extension. The partial messages extension gives the application precise control on exchanging message metadata and exchanging parts of messages. Unlike normal Gossipsub messages, Partial Messages are not opaque and peers can exchange message metadata separately from message data. An example use case is reconcilling when peers have some or all of a message. This works in a backwards compatible way. Notable changes: - New FanoutOnly topic option (#676). - WithMessageFilter to filter messages early in the notification pipeline (#678). Fixes & improvements: - Structured RPC logging - Fix leaked state with OnNewIncomingStream/OnClosedIncomingStream (with failing test first) - Renamed {Add/Remove}Peer β On{New/Closed}OutboundStream - partialmessages: init fanout if empty (#690) - Log errors when handling RPC in extensions (#668) - Ensure Hello Packet is first RPC sent; set write deadline for outgoing messages - Added Rpc.From() - Threadsafe dynamic direct peer management for GossipSub (#673).
This release introduces support for the partial messages extension. The partial messages extension gives the application precise control on exchanging message metadata and exchanging parts of messages. Unlike normal Gossipsub messages, Partial Messages are not opaque and peers can exchange message metadata separately from message data. An example use case is reconcilling when peers have some or all of a message. This works in a backwards compatible way. Notable changes: - New FanoutOnly topic option (#676). - WithMessageFilter to filter messages early in the notification pipeline (#678). Fixes & improvements: - Structured RPC logging - Fix leaked state with OnNewIncomingStream/OnClosedIncomingStream (with failing test first) - Renamed {Add/Remove}Peer β On{New/Closed}OutboundStream - partialmessages: init fanout if empty (#690) - Log errors when handling RPC in extensions (#668) - Ensure Hello Packet is first RPC sent; set write deadline for outgoing messages - Added Rpc.From() - Threadsafe dynamic direct peer management for GossipSub (#673).
This release introduces support for the partial messages extension. The partial messages extension gives the application precise control on exchanging message metadata and exchanging parts of messages. Unlike normal Gossipsub messages, Partial Messages are not opaque and peers can exchange message metadata separately from message data. An example use case is reconcilling when peers have some or all of a message. This works in a backwards compatible way. Notable changes: - New FanoutOnly topic option (#676). - WithMessageFilter to filter messages early in the notification pipeline (#678). Fixes & improvements: - Structured RPC logging - Fix leaked state with OnNewIncomingStream/OnClosedIncomingStream (with failing test first) - Renamed {Add/Remove}Peer β On{New/Closed}OutboundStream - partialmessages: init fanout if empty (#690) - Log errors when handling RPC in extensions (#668) - Ensure Hello Packet is first RPC sent; set write deadline for outgoing messages - Added Rpc.From() - Threadsafe dynamic direct peer management for GossipSub (#673).
This release introduces support for the partial messages extension. The partial messages extension gives the application precise control on exchanging message metadata and exchanging parts of messages. Unlike normal Gossipsub messages, Partial Messages are not opaque and peers can exchange message metadata separately from message data. An example use case is reconcilling when peers have some or all of a message. This works in a backwards compatible way. Notable changes: - New FanoutOnly topic option (#676). - WithMessageFilter to filter messages early in the notification pipeline (#678). Fixes & improvements: - Structured RPC logging - Fix leaked state with OnNewIncomingStream/OnClosedIncomingStream (with failing test first) - Renamed {Add/Remove}Peer β On{New/Closed}OutboundStream - partialmessages: init fanout if empty (#690) - Log errors when handling RPC in extensions (#668) - Ensure Hello Packet is first RPC sent; set write deadline for outgoing messages - Added Rpc.From() - Threadsafe dynamic direct peer management for GossipSub (#673).
This release introduces support for the partial messages extension. The partial messages extension gives the application precise control on exchanging message metadata and exchanging parts of messages. Unlike normal Gossipsub messages, Partial Messages are not opaque and peers can exchange message metadata separately from message data. An example use case is reconcilling when peers have some or all of a message. This works in a backwards compatible way. Notable changes: - New FanoutOnly topic option (#676). - WithMessageFilter to filter messages early in the notification pipeline (#678). Fixes & improvements: - Structured RPC logging - Fix leaked state with OnNewIncomingStream/OnClosedIncomingStream (with failing test first) - Renamed {Add/Remove}Peer β On{New/Closed}OutboundStream - partialmessages: init fanout if empty (#690) - Log errors when handling RPC in extensions (#668) - Ensure Hello Packet is first RPC sent; set write deadline for outgoing messages - Added Rpc.From() - Threadsafe dynamic direct peer management for GossipSub (#673).
Summary
Adds a
WithSelfNotification(enabled bool)subscribe option that allows subscribers to opt out of receiving messages published by the local node. This addresses a common use case where applications need to avoid processing their own published messages.Closes #537
Changes
WithSelfNotification(enabled bool)as a newSubOptforTopic.Subscribe()false, messages published by the local host are filtered out innotifySubsbefore deliveryTestWithSelfNotificationcovering:Design
The filtering happens at the
notifySubslevel rather than inNext(), so self-published messages never enter the subscription channel buffer. This avoids wasting buffer space and prevents unnecessary drops when the subscriber is slow.Each subscription stores the local host's peer ID at creation time and compares it against
msg.ReceivedFromduring delivery.