I made brief mention of NILFS on my follow up article (http://www.kev009.com/wp/2008/12/more-linux-file-systems/). The shortlist: it is a log structured FS for better or worse. It offers much of the features of ZFS (which I think is log-influenced) and Btrfs. You probably haven’t heard of it because the TODO list is quite long (nilfs.org) and the devs and NTT (company behind it) haven’t been doing much of the social work required to get code reviewed, tested, improved and ultimately merged upstream. According to the Wikipedia article, log file systems should be best on SSD storage where seeks are less of a problem.

Indeed, NILFS could be a sleeper and add another contender to the ext4-Btrfs-Tux3 trifecta that is shaping up right now.

Both iSCSI and NFS have some problems. The Solaris iSCSI target is very buggy, but sounds like it may be workable for some people on the list. One advantage over NFS is that you can export zvol’s directly to the VM guests, so you can use ZFS to snapshot and clone the guest volumes rather than ESX. ZFS might be faster at destroying snapshots than ESX, and it also has some primitive replication ability through ‘zfs send’ and ‘zfs recv’, but all this needs testing before you count on it.

NFS is substantially less buggy than iSCSI, but it still has problems. I’ve problems where one failed/failing zpool will lock up NFS service for the whole system, not just for the pool that’s failing.

I think speed is not a clear win on either side. NFS might be about as fast as iSCSI for big files—it’s quite slow when there are lots of tiny files being opened and closed, but for big files, it sounds like sites are always migrating in one direction or another for who knows what reason. So few people use this stuff, you can’t really count on anything until you test it yourself.

I’m not sure what Andrew is talking about with the “half the size” comment. ZFS can make sparse volumes if you use the iSCSI approach where you get a volume taking no space from the pool filled with zeroes, and it starts to take pool space as the guest writes things other than zeroes into it, but (a) VMWare can do that natively/over-NFS with thin-provisioned volumes, analagous to the usual non-flat .vmdk’s in Workstation, (b) by default zvol’s get a reservation for their whole size so they don’t really take up less space unless you disable this and allow overcommitting, and (c) this doesn’t mean less I/O—the zeroes don’t have to be read or written, whether you allocate space for them or not. Andrew might have been thinking of ZFS’s compression feature, butif this is okay with you, why not enable filesystem-level compression on the guest and save the ESX-to-ZFS bandwidth? There’ve been some reports on the list of choppy non-realtime-friendly behavior with ZFS’s gzip compression, and gradual slowdowns to virtual lockups over a few days. The less-tight lzjb compression has less problem reports. both I think both are not well-tested with ESX so I’d wait for some real positive reports before counting on them to work well in a big vmware setup. What’s usually wanted for big vmware setups is deduplication, which ZFS doesn’t do yet.

if you want desktop virtualization you can use VirtualBox instead of vmware, under SXCE. just be sure to get 64-bit CPU for the extra kernel address space, and loads of RAM.

It does not degrade to a read-only mode on certain kinds of errors—I’m not sure from what other blog you picked up that idea, but think maybe you’re thinking of ext3, or maybe solaris’s UFS.

but in general, I think you are getting lost in bulletted-feature-list arcania. It doesn’t matter how many copies of some piece of metadata exist when the ZFS code which exists right now declares the whole pool corrupt and refuses to read any of the copies, which is what it often does. It’s also common to get in a situation where the pool seems to be working ok, reporting and recovering from errors, but if you reboot you’l lnever be able to import that pool again. Given the ZFS format’s well-advertised sorts of spatial redundancy it would be possible to write a really aggressive fsck tool or copy-out forensic tool, but THAT TOOL DOES NOT EXIST RIGHT NOW so it is dangerously misleading to write about ZFS’s resilience as if it can do anything the disk format could do assuming perfect software that we don’t have—we don’t approach anywhere near such software fantasies, how many years, two, three, four? after ZFS’s original release in Nevada.

The simple user interface is also a liability in some situations. If you’ll have a look at the zfs-discuss list you’ll see a variety of problems—different kinds of operation are called “resilver” not just device recovery, the output of ‘zpool status’ is often out-of-date or less informative than you can get by observing zfs’s internal status state machine indirectly, certain kinds of status like mirror dirtyness don’t survive reboot and should, it’s impossible to resilver two devices at once which is a problem for really big pools or for certain kinds of pool gymnastics, someone has a pool with a device that can’t be replaced because he interrupted a device-replacement half way through, many different errors are compressed to the phrase “no valid replicas”, and the ONLINE/DEGRADED/FAULTED/OFFLINE statuses of devices are also compressed from a larger number of true internal statuses, hiding information in the name of simplicity.

A few serious bugs are still open, and many serious bugs were fixed recently and are not backported to Solaris 10 stable version. There seems to be a lag of almost a full year in backporting Nevada/Opensolaris fixes to plain Solaris, and they’ve stated they will only backport fixes when a Sol10 contract-holder complains about that specific problem, not proactively.

The original cause of the routine corruption from bouncing iSCSI targets is only specilatively explained and is *NOT* fixed. i mean, in that scenario, it’s (speculatively) a bad iSCSI stack causing the corruption, not necessarily ZFS bugs (maybe bugs in responding to the failure like not resilvering enough, but maybe not), but what’s sure is that other filesystems consistently deal with it with more grace than ZFS, so ZFS needs more robustness here. This is not fixed. And there are probably other things making pools non-importable.

Bob Paulson I’m using SXCE (solaris express community edition), which is like OpenSolaris but is a larger DVD, is not redistributable, comes in a SPARC version, works with LiveUpgrade instead of IPS, and has source code for a smaller percentage of the binaries delivered tha nOpenSolaris. Your best bet might be Nexenta, because they roll their own stable releases outside Sun, and they’re better than Sun at proactively backporting ZFS bug fixes from Nevada to their stable release, and you get most of the source. but I haven’t tried it myself, and I heard some goofy restrictiveness about the NexentaStor license that I don’t understand. ZFS is also in Mac OS X, FreeBSD, and Linux (FUSE), but I’m not sure it’s a good idea to use it there because of the number of bug fixes going in such that even Solaris 10 ZFS is too old IMHO—you may rightly want the absolute latest version, for bug fixes not features, so SXCE OpenSolaris and Nexenta will all be newer than Mac/BSD/Linux.

ZFS is not all bad, and I’m not telling people to stay away from it. My message is (1) backups, as in on another filesystem, are MANDATORY with ZFS, and (2) I’m tired of reading overblown hype chatter on blogs about ZFS, people with minimal experience telling each other how they heard it was supposed to behave so marvelously. Yes, ZFS still has hope, but it is currently right NOW much more corruption prone than UFS, ext3, XFS, and it is not getting more stable very quickly.

PPPS
You should probably learn how to use CSS to style your bullets instead of whining about it.

Especially:
list-style-type
list-style-position
padding
margin