Skip to content

Add options for record count and timeout for resolving DHT paths - #4733

Merged
whyrusleeping merged 8 commits into
ipfs:masterfrom
dirkmc:feat/namesys-value-count
Mar 23, 2018
Merged

whyrusleeping merged 8 commits into
ipfs:masterfrom
dirkmc:feat/namesys-value-count

Conversation

@dirkmc

@dirkmc dirkmc commented Feb 23, 2018 •

Copy link
Copy Markdown
Contributor

PR for issue #4723


EDIT: ~Kubuxu
Fixes #4723

License: MIT
Signed-off-by: Dirk McCormick <dirkmdev@gmail.com>
@dirkmc

dirkmc commented Feb 23, 2018

Copy link
Copy Markdown
Contributor Author

This PR adds two new parameters to ipfs name resolve and ipfs resolve:
ipfs name resolve <hash> --dht-record-count=5 --dht-timeout=20

Currently when resolving 16 IPNS records will be retrieved from the DHT (where the DHT k-value is 20). The dht-record-count parameter allows clients to retrieve less IPNS records from the DHT when resolving.

Currently the timeout for retrieving a public key for an IPNS record is one minute, and the timeout for retrieving the value itself is one minute. The public key and value are retrieved sequentially so it can take up to 2 minutes to time out. The dht-timeout parameter allows clients to set a timeout for retrieving both values (eg maximum of 30s to retrieve both).

License: MIT
Signed-off-by: Dirk McCormick <dirkmdev@gmail.com>
@dirkmc

dirkmc commented Feb 27, 2018

Copy link
Copy Markdown
Contributor Author

@vyzo @Stebalien @whyrusleeping could I ask you guys to take a look at this?
Thanks,
Dirk

Comment thread core/commands/ipns.go Outdated
opts.Depth = 1
}
if rcok {
opts.DhtRecordCount = uint(rc)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can get rid of this cast once ipfs/go-ipfs-cmdkit#14 is approved

@vyzo vyzo left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You might want to make a new method in namesys that takes the parameters, so that you don't have to rewrite all the references.

@dirkmc

dirkmc commented Feb 28, 2018

Copy link
Copy Markdown
Contributor Author

My concern was that we would then have three different methods for doing the same thing. Do you think that's not worth worrying about?

@vyzo

vyzo commented Feb 28, 2018

Copy link
Copy Markdown
Contributor

My suggestion was more to add the new method that takes the options, and redirect the old method to the new one with default options. That way you wouldn't change the existing code at all.

@dirkmc

dirkmc commented Feb 28, 2018

Copy link
Copy Markdown
Contributor Author

I understand, my concern is about having three different methods in the API where we only really need one. Is that not worth worrying about?

@vyzo

vyzo commented Feb 28, 2018

Copy link
Copy Markdown
Contributor

Hrm, dunno -- it's nice to have a simple method that takes default options.

@whyrusleeping

Copy link
Copy Markdown
Member

When in doubt, variadic options can be a good interface

@dirkmc

dirkmc commented Feb 28, 2018

Copy link
Copy Markdown
Contributor Author

Yes you're (both) right, that is nicer, I'll use variadic options

License: MIT
Signed-off-by: Dirk McCormick <dirkmdev@gmail.com>
License: MIT
Signed-off-by: Dirk McCormick <dirkmdev@gmail.com>
License: MIT
Signed-off-by: Dirk McCormick <dirkmdev@gmail.com>
@dirkmc

dirkmc commented Feb 28, 2018

Copy link
Copy Markdown
Contributor Author

Ok changed to use variadic options, let me know if that makes sense

Comment thread core/commands/ipns.go
name = "/ipns/" + name
}

output, err := resolver.ResolveN(req.Context(), name, depth)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

where did the depth arg go?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oh nvm, i found it

Comment thread core/coreapi/name.go Outdated
}

output, err := resolver.ResolveN(ctx, name, depth)
ropts := []nsopts.ResolveOpt{}

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i prefer writing this as:
var ropts []nsopts.ResolveOpt

License: MIT
Signed-off-by: Dirk McCormick <dirkmdev@gmail.com>

@whyrusleeping whyrusleeping left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't see anything wrong with this, pretty clean change overall.

I'd like to get one more πŸ‘ from someone else before merging though, maybe @vyzo or @Kubuxu

@vyzo vyzo left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why is the options package a separate package?
It would be nicer from a usability perspective if it was inside namesys so that we wouldn't have to import it separately.

other than that LGTM.

@dirkmc

dirkmc commented Mar 2, 2018

Copy link
Copy Markdown
Contributor Author

I put it in a separate package as a way of namespacing, so that you call nsopts.Depth(1) instead of namespace.Depth(1), so that it's clear those methods are working with options. If you think that's not an issue I'm happy to put it all in the top level namesys package.

@vyzo

vyzo commented Mar 2, 2018

Copy link
Copy Markdown
Contributor

Fair enough -- not really an issue :)

@Kubuxu Kubuxu left a comment •

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, using Opts pattern fits here very well.

@Kubuxu Kubuxu added the RFM label Mar 8, 2018
@Kubuxu Kubuxu added this to the go-ipfs 0.4.15 milestone Mar 8, 2018
Comment thread core/commands/ipns.go Outdated
cmdkit.BoolOption("recursive", "r", "Resolve until the result is not an IPNS name."),
cmdkit.BoolOption("nocache", "n", "Do not use cached entries."),
cmdkit.UintOption("dht-record-count", "dhtrc", "Number of records to request for DHT resolution."),
cmdkit.UintOption("dht-timeout", "dhtt", "Timeout in seconds for DHT resolution. Pass 0 for no timeout."),

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So, I might be missing something but IPFS commands have a timeout flag (e.g., ipfs --timeout=30s). And we can always set a DHT resolution timeout by setting a timeout on the context. What's the usecase for the additional timeout?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The use case is that with --dht-timeout the DHT collects as many values as it can within the timeout, and then returns the best value it has found. The global IPFS command --timeout will simply kill the command:

> ./cmd/ipfs/ipfs name resolve --timeout=5s QmSgxPeqLrnM1ZB1pno3tYMQfqLB4cWXuAQNfSWsk77apK
Error: Post http://127.0.0.1:5001/api/v0/name/resolve?arg=QmSgxPeqLrnM1ZB1pno3tYMQfqLB4cWXuAQNfSWsk77apK&encoding=json&stream-channels=true&timeout=5s: context deadline exceeded
> ./cmd/ipfs/ipfs name resolve --dhtt=5 QmSgxPeqLrnM1ZB1pno3tYMQfqLB4cWXuAQNfSWsk77apK
/ipfs/QmUNLLsPACCz1vLxQVkXqqLX5R1X345qqfHbsf67hvA3Nn

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe it would be better for the --dht-timeout parameter to take a string though, eg --dht-timeout=5s instead of assuming seconds eg --dht-timeout=5

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see. It would be nice to explain that in the help but I can't think of a succinct way of putting it...

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@dirkmc it will be better for it to parse the time.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added a commit to parse the time, and changed the parameter description to
Max time to collect values during DHT resolution eg \"30s\". Pass 0 for no timeout
That's the most succinct way I can think of to say it

dirkmc added 2 commits March 21, 2018 22:58
…olve

License: MIT
Signed-off-by: Dirk McCormick <dirkmdev@gmail.com>
License: MIT
Signed-off-by: Dirk McCormick <dirkmdev@gmail.com>
if err != ErrExpiredRecord {
t.Fatal("ValidateIpnsRecord should have returned ErrExpiredRecord")
_, err = resolver.resolveOnce(ctx, id.Pretty(), opts.DefaultResolveOpts())
if err == nil {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why arent we checking for a specific error anymore?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The resolveOnce() method used to call routing.GetValue() but now we call routing.GetValues() directly.
The mock ValueStore implementation's GetValue() method would get a single record and call VerifyRecord() on it, which would return ErrExpiredRecord/ErrSignature etc. We now call the mock ValueStore implementation's GetValues() method, which gets multiple records, calls VerifyRecord() on each one individually, and adds the record to the list of valid records if there's no error. If there are no valid records it returns routing.ErrNotFound.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I looked into whether I can check for routing.ErrNotFound but the go-ipfs-routing mock returns datastore.ErrNotFound instead of routing.ErrNotFound when it doesn't find a key:
https://github.com/ipfs/go-ipfs-routing/blob/master/mock/centralized_client.go#L46

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Alright, that seems fine. Thanks for the response!

@whyrusleeping whyrusleeping left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, just one question on a test decision, then i'll merge.

Thanks @dirkmc :)

@Kubuxu Kubuxu added need/author-input Needs input from the original author and removed RFM labels Mar 23, 2018
@whyrusleeping
whyrusleeping merged commit 41d82ee into ipfs:master Mar 23, 2018
@dirkmc
dirkmc deleted the feat/namesys-value-count branch March 23, 2018 17:45
hacdias pushed a commit to ipfs/boxo that referenced this pull request Jan 27, 2023
Add options for record count and timeout for resolving DHT paths

This commit was moved from ipfs/kubo@41d82ee
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

need/author-input Needs input from the original author

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants