ESXi 5.1 - OS choices for a File Server

Question

This past black friday i put together a small box I intended to use for a soho VM lab.

• i5 processor
• B75M-D3H Gigabyte motherboard
• 16GB of RAM
• 1TB seagate constellation HD (SATA3)

I want to run a simple file server inside a VM on it. Right now everything is sitting on the 1TB seagate HD, but my plan is to take a 300GB HD from an existing computer and use that to hold the ESXi hypervisor and various VM images. I need the esx server for me to easily bring up images for work. The 1TB will be used as storage for the file server

I downloaded FreeNAS and NAS4Free and installed them to seperate VMs. Using FreeNAS and NAS4free I gave them 100GB Hard Drives and had them formatted as ZFS. I then shared them out via both CIFS/SMB and NFS. My environment is a mix of PC, Linux and Mac.

I'm on a Gigabit LAN. I started doing some simple tests. From a windows 7 box I just started to read/write files of varying sizes (10MB, 400MB, 1GB). Without going into a lot of detail, CIFS is generally beating the snot out NFS. On Avg, CIFS is hitting ~80MB/s and NFS ~25MB/s. On NAS4Free NFS was getting like ~10MB/s

One note about the VMs, on both I was able to install the VMXNET3 drivers and so they are both using that NIC.

My goal is to have relatively fast and stable file server with low administrative overhead. I realize I'm not going to break any performance records running the VM this way. So the questions are:

1. Are those speeds typical? I feel like I should be getting like 100MB/s average.
2. Am I crazy for tyring to run FreeNAS/NAS4Free on a VM? Should I just use Windows or a distro like Ubuntu? Feels like a lot of overhead so far.
3. Is there anything I can do to tweak ESX, FreeNAS/NAS4Free, NFS, ZFS to get better throughput
4. Is there a better configuration given what i'm trying to do and the hardware i have?

I haven't included more information as it seems easier to just answer whatever questions people might have about my setup.

Thanks in advance.

Answer 1

Are those speeds typical? I feel like I should be getting like 100MB/s average.

You didn't specify how many drives you have, other than 1TB (SATA3). You could theoretically get some more.. juice by doing RAID10 (4 drives; 2 pairs mirrored and then both pairs striped). That alone (no real configuration tweaks) might squeeze out a better reads for all types you mentioned.

Am I crazy for tyring to run FreeNAS/NAS4Free on a VM? Should I just use Windows or a distro like Ubuntu? Feels like a lot of overhead so far.

No, you're not crazy. In fact, I asked the same question on ServerFault. You're not crazy for trying to run those distros on a VM. There are however a crap load of caveats in doing so especially in a VM and especially depending on the VM hypervisor. Using Windows or Ubuntu would probably be simpler, or rather, FreeNAS/NAS4Free unvirtualized would be a lot easier. Not having to deal with the quirks of each hypervisor and how it treats its guests or what direct resources the guests can have can be a nightmare. So far I've tried with VMWare ESXi 4.1U3, 5.1 (patched) and Hyper-V 2012 (v3). I managed to get it working on 5.1 (patched) but it took a lot of time and effort just to get it all working correctly. I'm going to blog about this as I feel like I just went to Mordor to drop off the one ring.

Is there anything I can do to tweak ESX, FreeNAS/NAS4Free, NFS, ZFS to get better throughput

I think my first answer covers this, but I'll address specifically ESX, ZFS and FreeNAS as that's my exact setup.

• ESXi: Your best bet for performance with ZFS in FreeNAS is to give direct access to the disks. In hypervisor lingo, this is referred to as passthrough (or PCI-passthrough in VMWare). This means that FreeNAS can access each disk it's going to use without using a vmdk file within ESXi; this ultimately means better/faster performance. After reading on and on about ZFS from numerous sources, it's clear that for ZFS, less is more. Don't get a fancy RAID card. That won't help with redundancy or performance. ZFS likes unencumbered, direct disk access. SAS HBA (Host Bus Adapters) in JBOD mode are pretty much the way to go. VMWare's HCL for SAS HBAs is a little limited but the FreeNAS community is pretty good for finding compatible cards and there's also http://ultimatewhitebox.com that may be of some help.
• ZFS: I'll refer you to this guy: http://constantin.glez.de/blog/2010/01/home-server-raid-greed-and-why-mirroring-still-best. His article explains why mirroring is better than doing RAID-Z. I used to be of the RAIDZ/5/6 camp but after reading this article and thinking things over a bit, I've switched to mirroring and striping (RAID10/01). It's easier and faster.
• FreeNAS: I love FreeNAS 8.3. It's a solid distro. I know little to nothing about FreeBSD but I'm working on that. Running FreeNAS in a VM is do-able, but once you introduce HBAs into the equation, things get a little dicey. In my case, I ran into a few problems. Namely, getting FreeNAS to view the disks from the LSI SAS HBA was painful. It wasn't until I found a post in a forum post that stated for VMWare ESXi 5.1, you need to apply 3 patches from VMWare to fix passthrough I was able to move forward. After applying the patches to ESXi I was able to get the disks to be seen in FreeNAS, I had to deal with the VMXNET3 issue. Luckily, I found a great guide for installing the VMWare Tools in FreeNAS. After that, I encountered the lovely IRQ interrupt storm (great error name, right?) - of which there's no 100% clear solution. I managed after several hours of troubleshooting to figure out I had to disable ACPI on the BIOS of the host. Welcome to vmguest troubleshooting hell. (Keep in mind this was my last attempt which was successful; I bailed on VMWare ESXi 4.1U3 and Hyper-V 2012 core). I'd estimate that I spent about 24+ hours in trying to get this to work (total for all 3 hypervisors).

Is there a better configuration given what i'm trying to do and the hardware i have?

Well, from reading your parts list, I'd argue you'd need more disks and probably a compatible SAS HBA to give your FreeNAS VM direct access to your disks. That will improve performance however, how much better your mileage may vary. Getting VMXNET3 working is a big boost to network performance; avoid the E1000 as that will lower your network throughput.

TL;DR

• FreeNAS with ZFS native is far, far, far, far easier to do than as a VM guest.
• If you must use FreeNAS in a VM, you're obligated to investigate all the hardware aspects that come into play:
  1. CPUs (AMD-V, Intel VTx)
  2. Chipset virtualization support (IOMMU, AMD-Vi, Intel VTd)
  3. I/O virtualization (IOV/SR-IOV)
  4. Network virtualization (VT-c)
  5. SAS HBAs and compatibility.
  6. Don't forget about the hypervisor!
• FreeNAS may/will freak out about resources (IRQ interrupt storm to which there's no magic silver bullet) so your odds of having to do some troubleshooting is more than likely
• Avoid RAIDZ fancy ninja saving storage for faster, easier mirroring and/or mirror/striping
• There's no real 100% guarantee that this will all work even though it's all compatible. I'd also recommend therapy before, during and after your ordeal.

Feeling up to it after reading this? Maybe you are crazy. Hell, maybe I'm crazy.