I have a home server running Proxmox with a few VMs running. I currently have 5 shucked 8TB WD Ultrastar disks running in a TrueNAS VM using a m.2 to SATA adapter and I directly pass the adapter to the TrueNAS VM. This setup worked flawlessly until a few days ago.
Recently I moved apartment and upon booting the home server, only the first 3 of the 5 disks get detected by TrueNAS. If I swap any of the disks, it is always the first 3 that get detected. (I haven't tested for power but I doubt that this is the issue and I can explain below) Due to my RAID setup, this isn't much of an issue as I have 2 redundant disks and can recover the data if needed.
My questions
- Should I purchase a better m.2 adapter? If so which one?
- Should I just rearrange the setup to passthrough the motherboard SATA ports? This solution is potentially troublesome as I may accidentally passthrough the boot SSD or the disks could be passed through without it's vendor info (which is used to identify the disks so I can physically swap them around and TrueNAS doesn't loose track)
- Should I do a combination of the two, populate some/all of the motherboard SATA ports and purchase an M.2 to 2X SATA ports? I am not sure if each SATA port can be individually passed through, however, in Proxmox I can see 4 SATA controllers 2 grouped together with low PCIe address and another 2 grouped together with a much higher PCIe address. With limited testing I wasn't able to pass on board SATA controllers to TrueNAS.
- Also, if you had my hardware what would you change/upgrade given that I want a highly reliable, self sufficient system and I do mention some other small problems below.
Current Hardware
- Motherboard: X570 I AORUS PRO WIFI
- CPU: AMD Ryzen 5 5600G (I used to use a 3900X, downgraded to to needing GFX and lower power usage)
- RAM: 32 GB Corsair Vengeance (The RAM gets too hot to touch even in idle, should I get new ram? Possibly ECC)
- M.2 to SATA: JMB585 M.2 to 5 SATA ports
- Case: JONSBO N1 (it is pretty good but the front panel metallic piece makes a buzzing noise when using the disks and also limits airflow, so I removed it)
- Boot Drive: 2TB Samsung 860 EVO SATA
- Network: Mellanox ConnectX-3 CX311A (SFP+) 10GB
Hardware Arrangement
- PCIe slot: Mellanox card
- Front M.2: JMB585 adapter (every port is plugged in)
- Back M.2: not used
- SATA port 3 on motherboard is given to SATA SSD
- The rest of the SATA ports on the motherboard are not used
BIOS Setup (relevant stuff)
- SVM Mode [Enabled]
- IOMMU [Enabled]
- CSM [Disabled]
- Everything is can be considered default
Probable cause of only 3 disks out of 5 showing
JMB585 M.2 to SATA
The JMB585 M.2 to SATA has been perfectly reliable, however from my small investigation, this adapter likes to corrupt some data (which I believe is corrected by the RAID setup). Maybe the JMB585 is broken? I see these logs in the boot sequence sometimes.
The JMB585 adapter is too thin for me to hotswap the drives as it would flex too much. The PCB is thinner than a regular debit card and perhaps the adapter stopped working due to flexing.
Shared/limited PCIe lanes
The motherboard has an "issue" where if you use both m.2 slots, the m.2 drives will be given less PCIE lanes (and only the first 3 drives visible like it is currently).
However, after noticing this problem I switched the boot drive for the SATA one and it worked as expected. There is now nothing using the PCIE lanes except the internal IO, M.2 to SATA, Mellanox card and 1 SATA SSD. So there is no reason for the Adapter to fail.
Booting the system with everything unplugged except the Boot SSD and M.2 Adapter yields the same results.
This M.2 limitation is not in the manual and I only found out after searching the internet after setting up this system.
The other two disks might not be getting power
I doubt that this is the case because in normal circumstances, after passing the JMB585 adapter through to TrueNAS. Shutting down the TrueNAS VM shuts down the disks connected. When the host system is powered on, all the disks get powered on.
Currently shutting the TrueNAS VM with 3 of the 5 disks being connected does result in a quieter system but there is still the sound of disks running. So I assume that the first 3 disks are powered off and the 2 that were never connected remain in an idle powered on state.
Rebooting the system
When I had this home server in my previous apartment, I would reboot occasionally for various reasons and never ran into this issue again. Sometimes I would disassemble the system to clean and/or to do cable management. When transporting the server to my new apartment, it was wrapped in thick winter blanket and put in a suitcase and walked to the new apartment. The server showed no signs of damage during transport.
