You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
IPNS republisher should not rely on DHT to store (possibly expired) records #4749
The republisher relies on the DHT to store records, but those records may expire before republish occurs (eg if the republishing node goes offline). We should store them somewhere else in the datastore.
See #4742 (comment) for more details
Proposal: make the republisher wrap the namesys service in a republishing namesys service. When calling Publish on the RepublishingNamesys, it would store the record in a datastore and then continue to republish that record until replaced or deleted.
So, as I noted on that PR, the underlying namesys relies on this record itself to keep track of the sequence number.
I can think of three solutions:
Have the underlying namesys record the sequence number separately along with the last published value. IMO, we should do that regardless. Additionally, have the republishing nameservice record which names it should republish (and their last publish times?).
Have the namesys expose known, published records. That would allow us to keep the republisher simple. However, that also complicates the namesys Publisher interface (which, arguably, shouldn't expose this information).
Quick note: you should be able to update to latest-gx-tagged go-libp2p-kad-dht, go-ipfs-routing, go-libp2p-record for this. I should have left everything working on that front.
I noticed that the Pubsub publisher puts an IPNS record directly into the datastore, so that the republisher can later retrieve the record's sequence number. But it looks like namesys will always publish to both the DHT and to Pubsub anyway, so is it necessary for Pubsub publisher to do this?
And yes, i think the republisher should care about pubsub published records, as these will have a different seqno.
Also, it's not inconceivable that we do just pubsub publishing in the future.
Yes, that's an artifact of the implementation.
Ideally they would both use the same seqno, but it seemed too complicated at the time i was writing that code and opted for different seqnos.
This kept the implementation straightforward, as the two publishers could be kept completely separate and without changes to the extant publisher.
But since we are hacking the publishers, we might as well go the extra mile and have them use the same seqno for the record.
It shouldn't be causing any problems though.
Is there a case in which we would want to publish only to pubsub (but not to the DHT)? I'm wondering if the pubsub publisher should call out to the DHT publisher itself, which would take care of all the sequence number handling that the pubsub publisher has to do at the moment
Activity
dirkmc commentedon Mar 1, 2018
@whyrusleeping do you have a suggestion for how we should store these records?
Stebalien commentedon Mar 1, 2018
Proposal: make the republisher wrap the namesys service in a republishing namesys service. When calling
Publishon theRepublishingNamesys, it would store the record in a datastore and then continue to republish that record until replaced or deleted.Thoughts?
whyrusleeping commentedon Mar 1, 2018
That sounds like a reasonable proposal to me.
dirkmc commentedon Mar 1, 2018
Should we merge in the PR to use variadic options for the namesys interface first?
#4733
dirkmc commentedon Mar 1, 2018
@Stebalien is this what you were imagining? #4753
Note: the relevant commit is d68b5ce
Stebalien commentedon Mar 2, 2018
So, as I noted on that PR, the underlying namesys relies on this record itself to keep track of the sequence number.
I can think of three solutions:
hsanjuan commentedon Mar 2, 2018
Quick note: you should be able to update to latest-gx-tagged
go-libp2p-kad-dht,go-ipfs-routing,go-libp2p-recordfor this. I should have left everything working on that front.dirkmc commentedon Mar 7, 2018
I noticed that the Pubsub publisher puts an IPNS record directly into the datastore, so that the republisher can later retrieve the record's sequence number. But it looks like namesys will always publish to both the DHT and to Pubsub anyway, so is it necessary for Pubsub publisher to do this?
dirkmc commentedon Mar 7, 2018
And I guess a related question: Should the republisher care about records published directly to Pubsub (but not to the DHT)?
vyzo commentedon Mar 7, 2018
That's so that both itself and the DHT publisher can find it; the latter is important if you run your daemon having disabled pubsub.
vyzo commentedon Mar 7, 2018
And yes, i think the republisher should care about pubsub published records, as these will have a different seqno.
Also, it's not inconceivable that we do just pubsub publishing in the future.
dirkmc commentedon Mar 7, 2018
It looks like the pubsub record sequence number may be different from the DHT record sequence number for the same value in the following scenario:
I'm not sure if that actually causes any problems, but it's probably worth fixing
vyzo commentedon Mar 7, 2018
Yes, that's an artifact of the implementation.
Ideally they would both use the same seqno, but it seemed too complicated at the time i was writing that code and opted for different seqnos.
This kept the implementation straightforward, as the two publishers could be kept completely separate and without changes to the extant publisher.
vyzo commentedon Mar 7, 2018
But since we are hacking the publishers, we might as well go the extra mile and have them use the same seqno for the record.
It shouldn't be causing any problems though.
dirkmc commentedon Mar 7, 2018
Yes I agree, @Stebalien suggested simply comparing the values to make sure the sequence number is not incremented when the same value is published.
dirkmc commentedon Mar 7, 2018
Is there a case in which we would want to publish only to pubsub (but not to the DHT)? I'm wondering if the pubsub publisher should call out to the DHT publisher itself, which would take care of all the sequence number handling that the pubsub publisher has to do at the moment
vyzo commentedon Mar 8, 2018
Currently no, but we might want this in the future.
dirkmc commentedon Mar 8, 2018
I think if we don't always publish to the DHT it will cause an issue with publishing to a key shared between peers:
store IPNS records *outside* of the DHT
store IPNS records *outside* of the DHT
store IPNS records *outside* of the DHT
store IPNS records *outside* of the DHT
store IPNS records *outside* of the DHT
store IPNS records *outside* of the DHT
store IPNS records *outside* of the DHT
store IPNS records *outside* of the DHT