iSCSI for VAAI/ODX, or NFS, on a file server with multiple distinct shares

Question

I'm building a 3 server setup (FreeNAS, FreeNAS replication, ESXi) for a small network with a handful of users and a couple of "power users". The aim, briefly, is to migrate from multiple machines and windows file shares to more centralised and better resource usage, and higher quality and reliability.

The hardware is adequately specced (ESXi on octocore Xeon + Supermicro, and FreeNAS on quad fast Xeon for CIFS with a ton of RAM and Intel SSDs). But I'm expecting a considerable learning curve because of the differences of what I'm moving them from and to.

I was intending to run my file shares as NFS (for ESXi) and CIFS (mostly Windows + a couple of Mac clients). Then I came across VAAI and ODX and thought "wow", as the workloads look like they would really benefit from them. But as with any iSCSI, they'd replace NFS by one initiator/one target per share only unless one really loves corrupt data. So I'm wondering whether I have to give up on the idea of these nice shiney protocols, or whether I can get at least some benefits they offer if not all of them:

• If I ran ESXi (but not CIFS) as iSCSI, then the question becomes how to copy VM store files off the iSCSI pool if I wanted to, which will be useful, or indeed, how to upload extra files onto the pool. I don't want constant access but I would like to feel I have a way to access the iSCSI based store either read-only, or read/write when the usual initiator isn't connected.

• Is there a way to maintain a second iSCSI initiator running as read-only within FreeNAS itself, so that I can use FreeNAS itself to pull data that I know is idle (old files etc) from an iSCSI target on FreeNAS, to a separate machine?
  
  Ideally it would mean that when ESXi is disconnected, I can locally copy files to/from/between the VM store and an NFS share, or make the files on the VMFS store accessible and shared read-only to other machines

I understand that iSCSI doesn't 'know' about files, and that a file and its metadata which was in use wouldn't be guaranteed to be served correctly to a second 'interloper' read-only initiator if it was changed halfway through the request or something, but I'm thinking here about reading from old ZFS snapshots or when the main initiator isn't connected, or files known to be idle - situations where there may not be the usual risk of conflicting instructions and corruption, and for a small setup it would be helpful.

Answer 1

Why not try to squeeze every drop of performance from the fast disk arrays? I realize that, to some people, this performance benefit from using iSCSI with VAAI/ODX over simple NFS approach may be not so crucial, but I personally found iSCSI as the best way to achive maximum disks utilization. I've been testing that using Starwind vSAN acting as my storage array and ESXi host acting as an initiator. Most of all, I was pleasantly surprised when I migrated VMs with the VAAI feature enabled. It was around 2 times faster.

You could try Starwind vSAN as well. It supports both: VAAI and ODX, but unfortuantely their free version isnt allowed to do iSCSI, but you can request NFR license from them: https://www.starwindsoftware.com/starwind-nfr-license-users I did the same thing in my time.

Answer 2

What I'm seeing here is conflicting thoughts regarding high performance protocols and the desire to share data.

NFS is going to be the easiest and most manageable protocol to use here. Use iSCSI for dedicated storage, and use NFS for shared storage. If you want to share the files that would otherwise be on "dedicated" storage, then revise whether or not you wish it to be dedicated.

Since you want to share your datastores, put them on a shared protocol. NFS has pretty great performance, and it doesn't sound like it will make much of a difference for your use case to utilize iSCSI over it. Sure, numbers might be better, but overall experience will likely be unchanged.

All of this is subjective, as I can't capacity plan for you. I will say one thing in summary: Be lazy in a way that enables you to be more lazy in the future. Keep things simple unless they need to get complex to solve a real problem.