Skip to content

feat(fuse): Expose MFS as FUSE mount point - #10781

Merged
lidel merged 22 commits into
ipfs:masterfrom
gsergey418:feat/10710-mfs-fuse-mount
May 6, 2025
Merged

lidel merged 22 commits into
ipfs:masterfrom
gsergey418:feat/10710-mfs-fuse-mount

Conversation

@gsergey418

@gsergey418 gsergey418 commented Apr 10, 2025 •

Copy link
Copy Markdown
Contributor

@gsergey418

Copy link
Copy Markdown
Contributor Author

Core functionality should be ready, need to write some tests and change up the documentation.

@gsergey418
gsergey418 force-pushed the feat/10710-mfs-fuse-mount branch from a85c978 to 5c7fa80 Compare April 11, 2025 12:21
@gsergey418
gsergey418 force-pushed the feat/10710-mfs-fuse-mount branch from 222d4dc to d1b4cc1 Compare April 11, 2025 13:10
@gsergey418
gsergey418 marked this pull request as ready for review April 14, 2025 09:53
@gsergey418
gsergey418 requested a review from a team as a code owner April 14, 2025 09:53
@gsergey418

Copy link
Copy Markdown
Contributor Author

Tests ready now, PR should be good for review.

@guillaumemichel guillaumemichel left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I didn't review everything at the moment.

@gsergey418 were you able to test mounting /mfs and interact with that directory (read/write)?

I didn't manage to test myself, simply mounting /ipfs and /ipns (using master) broke fuse in my debian VM test setup.

Would you mind updating fuse.md to reflect these changes?

Comment thread cmd/ipfs/kubo/daemon.go Resolved
Comment thread core/commands/mount_unix.go Resolved
Comment thread fuse/node/mount_nofuse.go Resolved
Comment thread fuse/node/mount_notsupp.go Resolved
Comment thread fuse/node/mount_unix.go Resolved
Comment thread fuse/node/mount_unix.go Resolved
Comment thread fuse/mfs/mfs_unix.go Resolved
Comment thread fuse/mfs/mfs_unix.go Resolved
Comment thread fuse/mfs/mfs_unix.go Resolved
Comment thread fuse/mfs/mfs_unix.go Resolved
@gsergey418

Copy link
Copy Markdown
Contributor Author

I didn't review everything at the moment.

@gsergey418 were you able to test mounting /mfs and interact with that directory (read/write)?

I didn't manage to test myself, simply mounting /ipfs and /ipns (using master) broke fuse in my debian VM test setup.

Would you mind updating fuse.md to reflect these changes?

@guillaumemichel Yeah, it mounts and umounts successfully on my machine and I can interact with it like a normal filesystem (ls, cat, touch, mkdir, stat, etc...). Reading and writing seems to work correctly and keeps the files intact. I've changed the variable names like you requested and added documentation.

@gsergey418

Copy link
Copy Markdown
Contributor Author

I think it would be good to add some way of displaying the file's CID, perhaps through xattrs, if you want full MFS functionality.

@lidel

lidel commented Apr 22, 2025

Copy link
Copy Markdown
Contributor

Triage note: thank you, due to holiday season, we may be slower to respond, will look into this after tackling priority work for 0.35.0-rc1

@guillaumemichel

Copy link
Copy Markdown
Member

Thanks @gsergey418 for the PR.

I have tested this branch on a fresh Debian Bookworm install, and it seems that most operations work great in fuse. E.g files added with ipfs add --to-files=<filename> can be listed from fuse, and files added from fuse can be listed with ipfs files.

I think it would be good to add some way of displaying the file's CID, perhaps through xattrs, if you want full MFS functionality.

I didn't manage to display a file's CID using xattrs

$ xattr /mfs/myDir/hello.txt
[Errno 95] Operation not supported: b'/mfs/myDir/hello.txt'
$ xattr /home/guissou/.ipfs/config
$ ...

The latest command works on a file outside of MFS (but I guess the file doesn't have any xattr attribute).

Is this feature working yet?


It seems that in the mounted MFS (fuse), files are readonly. It is possible to override a file, but not to edit it. It that expected? What would it take for files to be mutable?

@gsergey418

gsergey418 commented Apr 25, 2025 •

Copy link
Copy Markdown
Contributor Author

@guillaumemichel Thank you for the review. I will look into it.

@gsergey418

Copy link
Copy Markdown
Contributor Author

@guillaumemichel The xattr was there, you just couldn't list available xattrs because the structures didn't implement the fs.NodeListXattrer interface. I've added the support for it, so it should work now.

$ xattr mydir/
ipfs_cid
$ xattr -p ipfs_cid mydir/
QmdzR7T9jCHxQbSLPoToQPmSEoud6wEeNrQdwwk2V2fy85 

@gsergey418

Copy link
Copy Markdown
Contributor Author

@guillaumemichel I've added some tests, but haven't run them yet, can't get it to work on my machine. I'll check it out in CI.

@gsergey418
gsergey418 force-pushed the feat/10710-mfs-fuse-mount branch from b1bfc4c to 7e50e5a Compare April 29, 2025 16:03
@guillaumemichel

Copy link
Copy Markdown
Member

It seems that the FUSE sharness tests haven't been run by the CI either.

@gsergey418

Copy link
Copy Markdown
Contributor Author

I think you need to set the TEST_FUSE env var to 1 in your CI environment.

@guillaumemichel

Copy link
Copy Markdown
Member

@galargh do you know how to set TEST_FUSE=1 to enable FUSE sharness tests?

@galargh

galargh commented Apr 30, 2025

Copy link
Copy Markdown
Contributor

@guillaumemichel Sure! Currently, we have it disabled here - https://github.com/ipfs/kubo/blob/master/.github/workflows/sharness.yml#L49

It was disabled historically, but I'm not aware of the reasons. It might be OK to just turn it on.

@gsergey418

Copy link
Copy Markdown
Contributor Author

Let's see if this does anything. t0300-mount.sh AFAIK is the only FUSE sharness test, I don't think it's worth inventing something new just to test this.

@gsergey418
gsergey418 force-pushed the feat/10710-mfs-fuse-mount branch from 1a9d8ef to 3f88595 Compare April 30, 2025 14:45
@gsergey418

gsergey418 commented Apr 30, 2025 •

Copy link
Copy Markdown
Contributor Author

I think that broke some other tests in different scripts, so maybe they would have to be fixed or removed. I'm going on holiday right now, so I've won't be working on this for the next few days.

@gsergey418
gsergey418 force-pushed the feat/10710-mfs-fuse-mount branch 3 times, most recently from 72005fe to 5a53051 Compare May 5, 2025 10:01
@gsergey418
gsergey418 force-pushed the feat/10710-mfs-fuse-mount branch from 5a53051 to c579d85 Compare May 5, 2025 11:00
@gsergey418

Copy link
Copy Markdown
Contributor Author

I couldn't get the FUSE tests in CI working, so probably they were disabled for a reason. I'm still going to keep those sharness tests and lib changes just in case though.

@guillaumemichel

Copy link
Copy Markdown
Member

@gsergey418 do you pass the FUSE sharness tests locally?

@gsergey418

Copy link
Copy Markdown
Contributor Author

@guillaumemichel No, they don't work either way

@guillaumemichel
guillaumemichel force-pushed the feat/10710-mfs-fuse-mount branch from 0e3010b to 63852d9 Compare May 6, 2025 10:56

@guillaumemichel guillaumemichel left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK! Thanks a lot for your contribution @gsergey418!

I am good with merging the PR. FUSE is currently an experimental feature, and we can iterate from this initial state if we want to improve MFS later on to support attribute storage for example.

@lidel do we want this PR in v0.35.0? If not we can wait before merging

@gsergey418

Copy link
Copy Markdown
Contributor Author

Awesome! :D

@lidel

lidel commented May 6, 2025 •

Copy link
Copy Markdown
Contributor

my todo for 0.35.0-rc1 prep

  • resolve merge conflicts
  • add warnings about mutable endpoint risks (undesired preload of lazy-loaded DAGs, flaky writes not ready for production)
  • see if we can easily fix sharness to pass run with TEST_FUSE=1
  • merge

@lidel lidel left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you. Did some testing and while writes are flakey, read-only works good enough for 0.35.0-rc1.

To set user expectations, added some caveats and warnings about experimental nature of this feature and fixed small bugs that broke TEST_FUSE=1 ./t0030-mount.sh -v sharness tests. They pass now.

Merging 👍

@lidel
lidel merged commit 7c844ba into ipfs:master May 6, 2025
@lidel lidel mentioned this pull request May 6, 2025
33 of 45 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

4 participants