Most of your points are correct, so I just focus on the rest:
Using an SLOG device for the ZIL does only help you with small synced writes, so it is pretty mandatory if you want to store virtual machines on it and pretty useless in most other home use cases, especially backups and streaming media. As you can always later on add it and remove it, you should start without and then add only if necessary.
L2ARC can increase your read performance, but it is slower than RAM, needs extra RAM and only helps if the same data is read. Again, bad for streaming a whole movie or music, but good if you host a website that is heavily accessed or have hundreds of users accessing file shares. Rule of thumb is: first max out your RAM (depending on your board 32, 64, 128 or 256 GB most likely), then think about L2ARC.
ZIL and L2ARC on the same device is usually not a good idea, as their needs are directly opposed:
- ZIL is written to constantly for small random synced IO (large and sequential IO bypasses it, async IO of any kind does not use it at all), which means you want an SSD with very low write latency (Intel is the only vendor I've found that specifies this characteristic even for the cheaper consumer SSDs), acceptable write IOPS (nearly all SSDs are sufficient here), and high amount of TBW so your SSD does not die each year from exhaustion. For size, < 10 GB is usually enough for small systems. Mirroring is preferred to prevent data loss when power and SSD fail at the same time.
- L2ARC on the other hand needs to be several times larger (> 64 GB is common, depending on RAM), is seldomly written to but read often, so you want high read IOPS, acceptable read latency and don't care about TBW that much. Mirroring is a waste of money in most cases, as it is only a cache device and can be lost and recreated without problems.
A single root pool is of course possible, but you save yourself some headaches if you mirror it. As it usually is hit not that much, two slow disks or even USB devices (each mainboard has at least two USB ports as headers internally) are perfectly fine for home use and you gain another usable disk slot. Especially when running without UPS two rpool devices really give you peace of mind.
Your pool size is correct, but it may be an option to go for 12 disks with either 2x Z2 (6 disks each) or 1x Z3 (12 disks each). As a rule of thumb, when using Z1/2/3, you should populate all your available disk slots at first, because while upgrading disk size one by one is trivial, adding more disks later on is impossible.
I don't know about Linux (should work fine), but have you looked at other illumos-based systems? OmniOS is small, simple and stable and can be customized to your needs (it also includes KVM and LX-branded zones). SmartOS is similar, but focused heavily on zones (containers), so you can run all your services independent from one another and can even run Linux guests in those zones for the few services that are not available on Solaris. There are also Delphix and NexentaStore Community Edition, but I have not tested them.
My personal suggestion:
- Use whatever operating system your are comfortable with (if you like the stability of Solaris, try OmniOS or if you want virtualization, try SmartOS)
- Use mirrored rpools on USB disks (USB3 HDDs or USB3 sticks with SLC memory) to expose more slots for disks
- Use 6 ports from the mainboards and 6 ports from the HBA card so you can lose a controller and your system keeps running
- 2 free ports can be used in the future for SLOG or L2ARC devices depending on your needs
- Layout (12 is a very nice number, 16 would be next best because most controllers are 4 or 8 port):
- If you need maximum performance: 6x2 mirrors, each on both controllers
- If you need maximum resilience: 4x3 mirrors or 1x RAID Z3(12) or 2x RAID Z2(6)
- If you need maximum space: 1x RAID Z2(12)
- Maximize RAM first, then anything else
Regarding your follow-up-questions from comments:
I like the idea of using two mirrored USB3 sticks for the system, but is it bootable ?
USB sticks are essentially the same as USB disks, so you can boot from them without problems (except on very old mainboards, but everything from the last ten years should be fine). Some systems like SmartOS or ESXi even advertise it as best practice.
Some on the other hand (like FreeNAS) do not recommend it, because they are not customized for USB sticks and therefore constantly write to the disks and wear out cheaper sticks pretty fast (this is why so many Raspberry Pis fail early - the Linux system thinks it has an indestructible HDD and not some 5 EUR USB stick or SD card that is designed for infrequent writes like from a digital camera).
With SLC sticks (or real SSDs), you do not have these problems. Of course, they are more expensive, about 30 to 40 EUR for 16 GB sticks (MachExtreme MX-ES are about the only worthwile things in this sector). SSD can be cheaper (30 EUR for 32 GB), but you would need a USB adapter and they take up more space. You can use them outside of the case for quick backups/swaps or inside for access control (read: if you have children who like shiny toys).
it seems I don't need log/cache disks, should I use all 14 ports for disks ?
Depends on your needs and budget. If you use mirrors, you are flexible to add them later. If you use RAID-Zn, I would set the final amount before creating the pool, because you cannot easily add more. On the other hand, you might want to keep some ports free for backup (using slot-in caddys for 3.5" drives, for example) or for cache purposes if your needs change. It is up to you if you value space more than flexibility (and it depends on you much expansion cards your hardware supports).
Something like 2x7 RaidZ2 striped together ? If I do this and the controller fails, The pool will be failed, but if I replace the controller with an identical one, will it run again ?
Yes, and it works even if you use another controller, because everything is done in software. You just need to add enough disks so that each vdev works, and you can bring the pool back online.
In your case, if your 8-port controller fails, you need to add five (7 - 2) of those disks to the system in any way (the 8th is expendable anyway, because the other six disks are still running), for example with a 4 port controller and a single USB disk (not recommended, just to show that the connection is basically meaningless).
Usually you just replace the controller with the same model, because you know that configuration works without problems and performance is sufficient (with 8 disks per controller, the price of the controller itself is pretty small in relation anyway.
And if I do this, can I grow one part of the pool without touching the other one ?
You can grow the vdevs separately, but only as long as the pool itself is online (meaning after replacement of the controller and resilvering of any errors). Take into consideration that if you grow it "unbalanced", your data will not be rebalanced later on if the other vdev is expanded, except for any new and modified blocks (copy on write does not reorder data on read). This should be no problem for your performance needs, but I thought I'd mention it for completeness.
