Skip to content

implement mark and sweep garbage collection - #1420

Merged
jbenet merged 5 commits into
dev0.4.0from
feat/mark-n-sweep
Jul 11, 2015
Merged

jbenet merged 5 commits into
dev0.4.0from
feat/mark-n-sweep

Conversation

@whyrusleeping

Copy link
Copy Markdown
Member

This PR implements mark and sweep garbage collection and completely removes the concept of indirect pinning.

@whyrusleeping whyrusleeping added the status/in-progress In progress label Jun 23, 2015
@whyrusleeping

Copy link
Copy Markdown
Member Author

not yet ready for merge, i forgot to take into account that after rebasing on dev0.4.0, pins are stored in the blockstore, so i have to make sure to mark those on GC.

@jbenet

jbenet commented Jun 25, 2015

Copy link
Copy Markdown
Member

RFCR?

@whyrusleeping

Copy link
Copy Markdown
Member Author

@jbenet Yeah, I could use some CR. not quite done with things, but the basics are there.

Comment thread pin/gc/gc.go Outdated

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

describe the algorithm in a comment here. it's important that for this sort of thing we have a record of what it should be conceptually. as often that may not match what it is.

@jbenet jbenet self-assigned this Jun 30, 2015
Comment thread pin/gc/gc.go Outdated

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

now that we have prometheus, might be interesting to maybe mark some of our buffer datastructures to see if any of them are responsible for bad memory spikes.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what would this look like in code?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

make a special buffer with a prometheus gauge or somethng?

@jbenet

jbenet commented Jul 1, 2015

Copy link
Copy Markdown
Member

my biggest comment here is that we must address what happens when things are pinned during GC. Currently, GC will defer "what keys to keep or delete" to both the pin state and the blockstore, neither of which mutex-es with GC. A pin or an add (which pins) during GC could be incorrect. ("lose user data" incorrect)

Suggested approaches below.

(easy) add a gc lock

Add a lock (similar to an RWLock) which mutexes between "GC" and "Adds+Pins". Meaning, it allows concurrent access of any add and pin operation, but blocms GC during them. And likewise blocks them during GC.

This will preseve fast operation during most uses, but will peg the node a bit during GC. Similar to how allocation + GC happens in many memory GCs.

I don't expect GC to be so common that it must be run so often that this lock becomes a problem. However, it may have weird effects when there is a node with automatic gc upon hitting a "gc threshold" which is very near to the total amount of data pinned.

(hard) add add/pin log

Similar to the lock, but instead of blocking add+pins, it will instead generate a (persistent) log of operations that can be replayed to ensure the adds+pins can be properly applied after gc ends.

This is much harder to write + test, but would make add/pins not block.

Comment thread pin/gc/gc.go Outdated

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

my impression is that without concurrency, this will sequentially do IO + mutate memory, losing lots of time in the context switches with the kernel. Perhaps one good hack might be to have 2+ concurrent goroutines calling AddDag? would need some mutex on the map, so that might actually be enough to kill the improvement... but it is IO + lots of syscalls that we might schedule better...

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we should worry about complicating things when we notice a perf hit because of this.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sgtm! πŸ‘

@whyrusleeping whyrusleeping mentioned this pull request Jul 1, 2015
23 of 44 tasks
@whyrusleeping

Copy link
Copy Markdown
Member Author

Yeah, my thoughts were to add a RWMutex to the blockstore where the only taker of the write portion of the lock is the GC routine. (the easy route you describe above). We can transition to the log later, it wont require a migration or anything so i'm fine taking the quick and easy route for now.

Comment thread merkledag/merkledag.go Outdated

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this way of doing it spawns a goroutine per node and keeps them all alive until they all get in. maybe a concurrent approach with channels may fare better.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah, i can make that better

@whyrusleeping
whyrusleeping force-pushed the feat/mark-n-sweep branch 2 times, most recently from 2dfda14 to 0149e9e Compare July 2, 2015 17:01
@whyrusleeping

Copy link
Copy Markdown
Member Author

@jbenet I improved fetchgraph to still use one goroutine per node fetched, but the goroutine exits as soon as that node is received, as opposed to waiting till all its children are fetched. This would be easier if you could select on an array of channels, i considered using reflection but decided pretty code > slightly more efficient code for now

@whyrusleeping

Copy link
Copy Markdown
Member Author

separating the locking code on the blockstore into a separate interface complicates things... the write_cache wrapper needs to now cast the blockstore its wrapping as the locking blockstore in order to provide the same interface

@whyrusleeping

Copy link
Copy Markdown
Member Author

I'm gonna YOLO-cast it:

func (w *writecache) Lock() func() {
    return w.blockstore.(GCBlockstore).Lock()
}

Comment thread blocks/blockstore/write_cache.go Outdated

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is this ok to return since *writecache isn't exported?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, this is the way we should be doing interfaces in go. that way, in the calling code, i can do either:

var bs Blockstore
bs = WriteCached()

OR

var gcbs GCBlockstore
gcbs = WriteCached()

without any ugly casts or failure conditions.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what i mean is getting into an annoying case where i need the real type of the thing but i don't have it -- maybe we should also export WriteCache? shrug it's fine i guess

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i'm fine with exporting WriteCache

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

πŸ‘

@whyrusleeping

Copy link
Copy Markdown
Member Author

some of the tests in t0080-repo.sh seem wrong... for example: https://github.com/ipfs/go-ipfs/blob/master/test/sharness/t0080-repo.sh#L47

That one is expecting all the child objects of the welcome docs dir to show up in the recursive refs listing.

@whyrusleeping

Copy link
Copy Markdown
Member Author

we should also take another look at https://github.com/ipfs/go-ipfs/blob/master/test/sharness/t0080-repo.sh#L93

because we arent storing things in leveldb anymore...

@jbenet

jbenet commented Jul 4, 2015

Copy link
Copy Markdown
Member

@whyrusleeping

some of the tests in t0080-repo.sh seem wrong... for example: https://github.com/ipfs/go-ipfs/blob/master/test/sharness/t0080-repo.sh#L47

That one is expecting all the child objects of the welcome docs dir to show up in the recursive refs listing.

Yeah, that looks incorrect to me too.

we should also take another look at https://github.com/ipfs/go-ipfs/blob/master/test/sharness/t0080-repo.sh#L93

because we arent storing things in leveldb anymore...

agreed, does it pass now?

@jbenet

jbenet commented Jul 7, 2015

Copy link
Copy Markdown
Member

@whyrusleeping

  • thought we'd split the blocks changes into their own PR before full GC?
  • also what's up with tons of commits from tv's pinning changes leaking in here? is it from the block changes + rebasing? could we apply -> -> ? otherwise, the changeset is enormous and very hard to check.
  • also, the tests didnt like you this roll (circleci had an outage and was resource constrained). i've re-rolled them.

@whyrusleeping

Copy link
Copy Markdown
Member Author

@jbenet the commits from the pinning stuff keep leaking in since this is based on dev0.4.0 and i keep rebasing that branch, so these commits dont match up and it looks weird. I'll fix.

@whyrusleeping
whyrusleeping force-pushed the feat/mark-n-sweep branch 2 times, most recently from 7580503 to 9202d50 Compare July 7, 2015 16:12
@whyrusleeping

Copy link
Copy Markdown
Member Author

Alright, i separated our the blocks and merkledag changes to pr #1453, merge that first. Then this PR will be a single commit

Comment thread core/commands/pin.go

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i believe this needs to be brought back. (with likely a mark and sweep implementation)

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no, its okay that this is gone. you can still specify that you want to see indirect keys, but we arent going to bother with counting them anymore. Thats too much hassle and gains us nothing.

License: MIT
Signed-off-by: Jeromy <jeromyj@gmail.com>

dont GC blocks used by pinner

License: MIT
Signed-off-by: Jeromy <jeromyj@gmail.com>

comment GC algo

License: MIT
Signed-off-by: Jeromy <jeromyj@gmail.com>

add lock to blockstore to prevent GC from eating wanted blocks

License: MIT
Signed-off-by: Jeromy <jeromyj@gmail.com>

improve FetchGraph

License: MIT
Signed-off-by: Jeromy <jeromyj@gmail.com>

separate interfaces for blockstore and GCBlockstore

License: MIT
Signed-off-by: Jeromy <jeromyj@gmail.com>

reintroduce indirect pinning, add enumerateChildren dag method

License: MIT
Signed-off-by: Jeromy <jeromyj@gmail.com>
License: MIT
Signed-off-by: Jeromy <jeromyj@gmail.com>
Comment thread core/corerepo/gc.go

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we should flip the params here:GarbageCollect(ctx, n)

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we've always had the core.* functions take the node as their first argument. You want to change this everywhere?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ah. even when there is a ctx involved? i guess that makes sense

License: MIT
Signed-off-by: Jeromy <jeromyj@gmail.com>
License: MIT
Signed-off-by: Jeromy <jeromyj@gmail.com>
License: MIT
Signed-off-by: Jeromy <jeromyj@gmail.com>
@whyrusleeping whyrusleeping added this to the IPFS 0.4.0 milestone Jul 11, 2015
@whyrusleeping

Copy link
Copy Markdown
Member Author

The tests look good. Just so many of those random failures...

jbenet added a commit that referenced this pull request Jul 11, 2015
implement mark and sweep garbage collection
@jbenet
jbenet merged commit 95df5a1 into dev0.4.0 Jul 11, 2015
@jbenet jbenet removed the status/in-progress In progress label Jul 11, 2015
@jbenet
jbenet deleted the feat/mark-n-sweep branch July 11, 2015 00:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants