Repository navigation
Add fixed-period and conditional repo GC - #1495
Conversation
|
I like this. π Suggestions:
|
f6028f6 to
0d079dc
Compare
|
|
2.. Although not necessarily for nix https://nixos.org/nixos/manual/sec-nix-gc.html |
4431faf to
7ed7f35
Compare
There was a problem hiding this comment.
this file/function is already a huge mess. we'll want to split this out.
- this stuff belongs in the
fs-repopackage. I think maybe we can pass an option all the way in as to whether to initialize with GC or not. (or even no option, assume always so.
There was a problem hiding this comment.
cyclomatic complexity check:
$ gocyclo -over 15 . | grep -v Godeps | grep -v _test
32 main daemonFunc cmd/ipfs/daemon.go:115:1
30 cli parseArgs commands/cli/parse.go:199:1
27 merkledag_pb (*PBLink).Unmarshal merkledag/pb/merkledag.pb.go:101:1
27 cli parseOpts commands/cli/parse.go:62:1
25 http Parse commands/http/parse.go:15:1
25 secio (*secureSession).runHandshake p2p/crypto/secio/protocol.go:112:1
23 corehttp (*gatewayHandler).getOrHeadHandler core/corehttp/gateway_handler.go:85:1
22 merkledag_pb (*PBNode).Unmarshal merkledag/pb/merkledag.pb.go:205:1
21 main run cmd/ipfswatch/main.go:49:1
20 http getResponse commands/http/client.go:156:1
19 fsrepo (*FSRepo).SetConfigKey repo/fsrepo/fsrepo.go:487:1
19 merkledag_pb (*PBLink).VerboseEqual merkledag/pb/merkledag.pb.go:638:1
19 merkledag_pb (*PBLink).Equal merkledag/pb/merkledag.pb.go:684:1
18 http (internalHandler).ServeHTTP commands/http/handler.go:74:1
16 main main cmd/ipfs/main.go:58:1
16 corehttp (*gatewayHandler).putHandler core/corehttp/gateway_handler.go:249:1
There was a problem hiding this comment.
OTOH, ticker and ctx timeout stuff shouldn't exist in fs-repo. This is daemon stuff.
I will put the func runGC() in daemon.go for now.
|
@rht solid first pass! π this is headed in the right direction. some comments above o/ |
a4704b0 to
6ca010a
Compare
|
@jbenet RFCR, things have been ironed out. |
There was a problem hiding this comment.
This is outside of daemonFunc because it is cleaner this way. But it is the other way around for mountFuse, initWithDefaults. Which one is preferred? With runGC's, there is less res.SetError(err, cmds.ErrNormal) and less complexity in daemonFunc.
3e54b16 to
05c2554
Compare
There was a problem hiding this comment.
this should be a log.Event, not fmt.Println
There was a problem hiding this comment.
There are several fmt.Print* being used (that's why I use it). Should they be replaced with event log as well?
There was a problem hiding this comment.
there's a few fmt.Println, sorry, those are very few and mostly just the addresses users see when starting up the daemon.
we've considered maybe printing out some of the events, kind of like an http server, but it will take some work to figure out what to show and what not to.
|
Rebased. Not sure if this is going to be used. |
|
@rht very much yes! thanks for rebasing. @whyrusleeping last time i asked you said you'd be ok with this merging before dev0.4.0. ok to merge still? |
|
@rht tests fail now :( |
There was a problem hiding this comment.
Because I removed a duplicate import here. Fixed.
License: MIT Signed-off-by: rht <rhtbot@gmail.com>
Add fixed-period and conditional repo GC
Fixed-period GC landed! Thanks @rht |
|
@rht I'm testing this on venus and can't get it to trigger GC, neither periodic, nor via "Datastore": {
"GCPeriod": "1m",
"Path": "/ipfs/repo/datastore",
"StorageGCWatermark": 90,
"StorageMax": "20G",
"Type": "leveldb",
"NoSync": true
}, |
|
In the log, there is no gc trigger at all? If all the added files are pinned, they won't be gc-ed. And there should be continued periodic err message |
|
It works in my case; I just reproduced the err message through |
|
I got ahold of logs and I even see the periodic one twice :) But isn't it supposed to start GC'ing? |
|
Pinned files don't get gc-ed. |
|
Maybe I'm just totally confused -- I thought this would periodically check for the watermark, and do the equivalent to |
|
ic, just discovered the err: |
|
I think I set it to err and not do gc because if pinned files are larger than storagemax, the gc will be repeated wastefully. |
|
Do you mean these lines? https://github.com/ipfs/go-ipfs/blob/20b06a4cbce8884f5b194da6e98cb11f2c77f166/core/corerepo/gc.go#L126-L134
The GC run won't be wasted if at least a few blocks aren't pinned. On the ipfs.io gateways for example, there are a few GB pinned blocks, and the rest just fills up over time while people request stuff. |
|
No, that refers to GCPeriod? Then #2010 should fix this. |
|
Eh yes sorry I marked the wrong lines - I meant the ones in maybeGC that you linked. Let's carry on in #2010 |

WIP sketch.
Address #972