Repository navigation
Performance enhancement for write: don't use Direct/IO #576
Description
Hi, I am using encfs on top of my own fuse filesystem (1fichierfs) and I noticed that encfs is very unefficient about writes.
This is how it writes for a "stream write" (like a copy with cp), the below log shows Write requests with (offset:size)
Write (0:8)
Write (8:1024)
Write (1032:1024)
Write (2056:1024)
Write (3080:1016)
Write (4096:8)
Write (4104:40)
Write (4104:1024)
Write (5128:1024)
Write (6152:1024)
Write (7176:1016)
Write (8192:8)
Write (8200:72)
Write (8200:1024)
Write (9224:1024)
Write (10248:1024)
Write (11272:1016)
Write (12288:8)
Write (12296:104)
Write (12296:1024)
Write (13320:1024)
Write (14344:1024)
Write (15368:1016)
Write (16384:8)
Write (16392:136)
Write (16392:1024)
Write (17416:1024)
Write (18440:1024)
Write (19464:1016)
Write (20480:8)
Write (20488:168)
Write (20488:1024)
Write (21512:1024)
As you can see, there are very small writes down to 8 bytes, and some overwrite at offsets 4104, 8200, etc...
On top of that, since my fuse fs implements flush and fsync for debug purpose, NO flush or fsync is ever called before close.
Small "writes" like that can only happen with Direct/IO, otherwise the kernel would have buffered at least 4096 bytes.
If you compare with a "normal" write pattern, not from encfs, but an un-encrypted copy, the written sizes will all be 4096 (by default) or growing bigger up to 128K with -o big_writes
Sorry, the code is so divided into pieces that I gave up searching where this open with Direct/IO is, but just assuming it is there from what I see on my fs.
I was wondering if there were a reason why encfs would write with Direct/IO. I could understand it if it tried to flush/fsync (which would be useless on some filesystems since my "cloud-based" fs does NOT support that beside outputting a debug message), but it does not even try to flush/fsync... so what's then the justification for Direct/IO?
The justification could be the "overwrites" we can notice, but that could easily be fixed with light internal buffering, since the pattern of writing shows that the overwritten zones never go beyond 1024 bytes.
So the enhancement would be:
- use "normal" write (not Direct/IO)
- if necessary, apply internal buffering to avoid "overwrites"
Note: if you point me to where the code is for this open/writing (there are so many small files, it's a hell to find it), I might try to propose a patch.
Note2: that is probably why several persons reported "bad write performance" with other fuse fs like exFAT-fuse (and possibly NFS). I have optimised my fs to workaround this inefficient write pattern. In fact it just buffers the writes and does an actual write (involving thread switch + FTP with curl) only when the buffer is filled to a certain amount (pipe pattern). That mitigates the inefficient writes from encfs... but still yields poor performance because encfs itself does a lot of kernel/userland switches with such small writes!
[Edit]: this could also be caused by the use of "pwrite" and not by Direct/IO. In which case some buffering could really enhance performance for "stream" writes (probably 90% of the use cases !)
Activity
benrubson commentedon Feb 22, 2020
Did you try setting your EncFS configuration with bigger blocks ?
For example using 4096-bytes blocks ?
Bylon2 commentedon Feb 23, 2020
Hi benrubson, and thanks for you response!
You realize that 4K is the standard write block the kernel+fuse/libfuse use. You can't really specify less!
Explicitly adding:
-o big_writes,max_write=131072does not make the slightest difference on the issue: same write pattern with chunks as small as 8 bytes!..Note that 131072 is 128k which is the maximum buffer used by kernel.
In fact the "crypto" part of encfs works very well, but there are still a lot of bugs and sub-optmisation in the code.
I get quite often "invalid message", possibly a bug replying to low-level message.
And also "splice error" although I explicitly start with no_splice (on read and write).
Without the no_splice, you can almost systematically reproduce the cockup just by reading a mp4 movie stored on the crypto system with VLC. Mysteriously it does not happen with mkv, possibly the read pattern from VLC with each format is different.
It is also quite badly handling underlying filesystem errors. And since I run it on top of my own fuse-fs connected to a "cloud" it happens that you get errors (network issues for instance) which most of the time would block or crash encfs.
Had I finished my own fuse filesystem, I'd do a complete rewrite keeping the crypto part! ;-)
On a positive note, the version compiled from this repo is far less buggy than the one on the Ubuntu repo !
benrubson commentedon Feb 23, 2020
EncFS works on 1024KB blocks by default.
When you configure it using expert mode, you can choose bigger blocks.
Look for Filesystem Block Size in the man page.
Using bigger EncFS blocks could help improving performance.
Bylon2 commentedon Feb 23, 2020
Ah this one. Ok I'll try that, but I suspect the 8 bytes and the overwrite are sort of key that are written to the file, and that won't change although the overall performance could be better.
Also unfortunately too, it would mean copying all my data to the new FS to benefit from it!
Bylon2 commentedon Feb 23, 2020
Ok, I have investigated as you suggested @benrubson
The "overwrite" we see in the first log, for example at offset 8200 where encfs writes 72 bytes then overwrites them at the same offset with 1024 bytes, is probably an "under optimisation" due to the "Block Mac Header" parameter that is by default in "paranoia" mode.
The 8 byte size writes are (most certainly, didn't check the code!) the "cryptographic checksum" of the block.
Indeed, doing a filesystem without the "Block Mac Header" and with 4096 as the block size is much better. There is only one 8 byte write at the start (key initialisation?) then only 4096 writes and no "overwrite".
The performance is then "normal" although it cannot benefit from -o big_writes (which really improves things!).
I am closing this report as the performance is indeed "normal"... well as it is with 4K block write which is the fuse default.
And thanks for the tip again @benrubson
There is probably still improvement to the algorithm with "Block Mac headers", at least to remove the overwrite, and also some buffering would help, but we have been warned in the "man" that "Block Mac Headers" comes with performance overhead!
I'll keep "Block Mac Headers" in by the way, in spite of the "performance penalty" since my files are stored "on the cloud", it is a good practice to protect from the host tampering with files...
Squashed 'vendor/github.com/google/benchmark/' changes from 5b7683f4.…
Squashed 'vendor/github.com/muflihun/easyloggingpp/' changes from 468…