Skip to content

add more bitswap task workers - #1091

Merged
jbenet merged 4 commits into
masterfrom
bitswap-perf
Apr 19, 2015
Merged

jbenet merged 4 commits into
masterfrom
bitswap-perf

Conversation

@whyrusleeping

Copy link
Copy Markdown
Member

No description provided.

@whyrusleeping whyrusleeping added status/in-progress In progress topic/bitswap Topic bitswap topic/perf Performance labels Apr 17, 2015
Comment thread exchange/bitswap/workers.go Outdated

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i dont think 4 is enough?

4 * 256k = 1MB. i think the workers have to wait for the other side so a lot of time may be spent waiting for requests. this will be particularly bad in LAN settings where your 4 workers got stuck sending to 4 remote peers, but everyone else locally who you could be sending to is stuck waiting.

What's the cost of these? just a goroutine queue? if so, i'd crank it up to 16/32.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we can also try 16/32 and see how it goes. if not good we can bring it back down.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe make this a tunable setting?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe make this a tunable setting?

πŸ‘ even for just being able to test. we could make this an env var first, and move it to config if we find it's very useful. (think this is a good path to avoiding cluttering the config with crap. start with env and only pull into config if needed)

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, ill make it an env var

Comment thread exchange/bitswap/workers.go Outdated

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe IPFS_BITSWAP_TASK_WORKERS to scope it?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

too much typing πŸ‘Ž

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@whyrusleeping we may have other task workers. the problem with these env vars is that if we change them later we break user space. IPFS_TASK_WORKERS is way too general.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IPFS_BITSWAP_TW
IPFS_BITSWAP_WORKERS
IPFS_BITSWAP_TASK_WORKERS

@whyrusleeping

Copy link
Copy Markdown
Member Author

RFM

@jbenet

jbenet commented Apr 19, 2015

Copy link
Copy Markdown
Member

@whyrusleeping we need to scope that var-- it's going to cause problems. want me to do it?

@whyrusleeping

Copy link
Copy Markdown
Member Author

i can do it.

changed IPFS_TASK_WORKERS to IPFS_BITSWAP_TASK_WORKERS
jbenet added a commit that referenced this pull request Apr 19, 2015
add more bitswap task workers
@jbenet
jbenet merged commit e817852 into master Apr 19, 2015
@jbenet jbenet removed the status/in-progress In progress label Apr 19, 2015
@jbenet
jbenet deleted the bitswap-perf branch April 19, 2015 20:29
@cryptix cryptix mentioned this pull request May 11, 2015
9 of 36 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

topic/bitswap Topic bitswap topic/perf Performance

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants