This blog is about the Google Summer of Code project "ZFS filesystem for FUSE/Linux"

Saturday, January 19, 2008

Status update

Some people have (quite understandably) requested me to post a status update, so here it goes:

The good news:

The project is not dead.

The code is being updated every month with new features and bug fixes from OpenSolaris.

Critical bugs, and especially any bug reports of corruption, will receive my full attention, so please report them if that happens to you. I am 100% committed to fix any such bugs, but I haven't received any such reports in quite a time. However, I'm not sure if this issue has been truly fixed simply because I've been unable to reproduce it.

I will make an effort to review and integrate any patches that I receive that improve zfs-fuse.

The bad news:

The beta release is quite old already, so I should definitely make a new release..

I haven't been able to work on improving zfs-fuse, except for the monthly updates and the occasional bug fixes.

I have a couple of features that I've started to work on but haven't had the time to finish. One of them is async I/O and the other one is automatic disabling of disk write caching with hdparm and sdparm.

That said, if you would like to use zfs-fuse without having to worry about your data safety, please do the following:

Use the latest development code, from the Mercurial trunk (very important)

I am reluctant to release a new version until these last 2 points are dealt with in a safe way, either by displaying flashing big red warnings along with setting the mixer volume to the maximum and playing sounds of sirens, or by making zfs-fuse handle it automatically (probably better).

I'm really happy using zfsfuse in few places (my external hard drive and disk for backups in one of my server!). I use latests versions from Mercurial repository and have only few problems: random crashes (probably out of memmory, i have ~ 200 file system on by backup disk), after them i have weired mount entries (need to export, import, umount, and mount). And not working nfs out of the box.

Also moving beetween datasets using mv is really coping data, and as so is slow :/

If I will have any reproductible bug i will report it!

Generaly stability is very good, more is to be done in performance tuning.

Thanks for the update. Like many im glad to see your still punching this bad. Now here is a left hook.. Just wanted to share that u need the zlib-devel package to use with trunk where as the 0.4x you didnt.

Nice work, it looks like we need more people using it. That means somebody needs to modify the Debian (for me!) and Ubuntu (for everyone else!) installers to do a bootable ZFS root install. That'll nab headlines and get everybody trying it.

Bootable ZFS will be a first-in-the-world, so it'd be a big boost for people to become generally aware that it can work.

I realize modding an installer is a lot of work, though. So in the meantime, how about an instruction guide on converting your existing Linux system to bootable ZFS? I'll +1 Digg it!

yow. having been burned by way too much raid and LDM, I have been dying to drink the ZFS Grape Kool-Aid for a while. Is there anyone who can give me a sense if it's practical to use Ubuntu 7.10 as a base. I don't need to boot off of zfs. I'm perfectly content to make the boot and root partitions old-school.

what is that about write cache being turned off. From the zfs documentation I get the idea that even with write cache everything is fine if the hard drive follows the flush commands.I do not know what percentage of harddrives do not do it but that should be considered broken hardware.just my 2 cents

I'm using ZFS on Ubuntu, running a 2 terrabyte Zpool array (5x500gb HD's). It's been running smooth and steady since I set it up back in September. I've run the servers through all sorts of nightmare scenarios, from overwriting random sectors to removing entire drives, and hard booting the machine repeatedly in the middle of extensive operations, and it's continued to function without difficulty. The only complaints I have are:

1) Performance is a bit slow. I understand that this is being worked on, and it's not a major factor anyway.

2) I get a strange error which seems to have no real impact. My file system is reporting 7326 data errors, however I have examined all of the files on the disk and not a single one shows any sign of corruption. When I try to get a list of the corrupted files using "zpool status -v", the list fails and results in a core dump.

So far the second problem is just annoying, and I'm happy to ignore it. I'm just curious whether anyone else has experienced something similar.

Hi!! I have 3 months with ZFS in Linux without problems, compression=on, and with critical data(Docs, music, images), in ArchLinux and Kubuntu Hardy. Some crashes caused by wireless, but the file system is every time consistent. This is a great proyect!! Greetings from Chile.

I've got a ZFS setup running now at the company I work for. Its running exclusively using external USB drives (cheap) but I may also resort to using cheap consumer NAS hardware and create a single large file for ZFS-fuse to use as part of a pool. The whole setup is running inside a XEN domU VM with exposed PCI for the USB controller. I tried OpenSolaris (Indiana) first and found that throughput was TERRIBLE (1MB/sec file copy) vs the also terrible but much better 3-6M/sec I am getting with zfs-fuse on a linux domU.

So far no big problems but there is definately a memory leak in the zfs-fuse daemon. Doing a few rsyncs of all my data to the ZFS array from the existing media tends to have zfs-fuse fall over in a heap, requring a zfs umount -a; zfs mount -a and restart of all processes using ZFS mounted files. Its quite annoying but that aside it seems very "data-stable". I've also tried to kill it as best I can with no luck!

Regarding my previous "memory leak" comment. I'm a little confused now. I built the latest trunk with debugging enabled and ran it via valgrind with full memory checking turned on. When memory is exhausted, processes just start to die - including zfs-fuse.

After an hour or so of heavy rsync activity, my 640MB of RAM filled and I killed it all to see where the leaks were.

Turns out each thread (16 of them?) had about 3MB-5MB of definately leaked data and stemming from zfsfuse_listener_loop (fuse_listener.c:240). Which wasn't really a problem given that only one of those buffers should ever exist per thread and together that only makes up 80MB of my 500MB or so of process memory.

So where did all the other memory go? Not sure yet. I guess its not "leaked" as much as its allocated in excess. It might be that running with 2GB of RAM (which I dont have) would work perfectly... Can anyone else shed some light on this or do any similar tests?

I'm now using zfs-fuse on a fileserver for 20 people doing design work. Its handling the load quite well but it does have some memory issues. For the time being I'm just restarting the zfs-fuse daemon every week but its not ideal. If I had more time I'd love to get more involved.

I don't think the project is dead, its just a one man part-time show. ZFS certainly isn't dead and the design of this project means that those changes can, relatively easily, be ported over to zfs-fuse. So when solaris gets something new, so do we. ;) In theory at least.

Maybe at some stage in the distant future I'll have more time to try to work on something like this. My experience so far with ZFS is nothing but great!

Project is not dead. BUT, I have a question (please reply to my email address rudd-o at rudd-o dot com, there's no option to subscribe to replies).

Why "disable disk write cache"? On Linux + SATA, fsyncs() are the barriers that ZFS uses to ensure consistent data on disk (one after the transaction phase 1, and one after the tree top gets rewritten), so the disk shouldn't reorder disk writes between those barriers.

I read that everywhere, and ZFS is supposedly designed to work correctly with write caches (in fact, ZFS turns the cache ON except in the only situation where one should turn write cache off is when the disk is shared with UFS volumes, as those are sensitive to disk write ordering and the fs does not issue transaction barriers).

Why then the recommendation to disable write cache if ZFS is designed to use it? Performance drops abysmally when doing that.

Please keep updating the code. I use it all the time and I would like to see you continue -- maybe even help you package it for Hardy.

By way of hopefully encouraging others to help by lowering the bar on participation (but certainly not trying to steal anyone's thunder), I've put zfs-fuse up on github at http://github.com/dagbrown/zfs-fuse/ .

I am, you know, deeply thankful for the efforts poured into porting ZFS to Linux via FUSE. But I really need help and I have not been able to work this by myself (especially the negative inode cache by VFS part) - Ricardo, is there any progress being made on this? Is the Sun situation affecting this project?

zpool create just hangs for me.. :( I'm using the mercurial repository on Ubuntu 2.6.24-19 amd64. Another interesting thing is zfs-fuse brings out a bug in htop where htop shows hundreds of sequential PID's of zfs-fuse running. ps does not show this.

I've been running ZFS with OpenSolaris for over a year now. One thing I haven't been able to understand is that once a drive is made into a ZFS pool, it can never be used for anything but ZFS again. Using dd, shred, and even a windows "low-level" formatting utility to overwrite the entire disk does not work. Does ZFS overwrite the disk's UBA area?

Also if anyone is confused how to disable write caching, use: hdparm -W 0 /dev/disk/by-id/{drive}