Skip to main content
Hardware Recommendations Beta

Return to Answer

added 8 characters in body
Source Link
Adam Wykes
Adam Wykes
  • 2.4k
  • 1
  • 11
  • 20

This setup depends on you not needing to saturate the PCI-E bus, much like in GPU bitcoin mining, where a 1x connection to each of the mining GPUs was often sufficient. Putting a splitter into each of the native PCI-E slots, you get 24 GPU connections at, I think, 1x PCI-E 2.0 each, for a total transfer rate (theoretical up+down) of 1000MB/s. However, there are only 40 lanes on each CPU, so unless the PCH on the motherboard takes care of that part of the issue (I'm not certain how PCH works), each GPU will get something more like 833 MB/s theoretical maximum bandwidth.

This setup depends on you not needing to saturate the PCI-E bus, much like in GPU bitcoin mining, where a 1x connection to each of the mining GPUs was often sufficient. Putting a splitter into each of the native PCI-E slots, you get 24 GPU connections at, I think, 1x PCI-E 2.0 each, for a total transfer rate (theoretical) of 1000MB/s. However, there are only 40 lanes on each CPU, so unless the PCH on the motherboard takes care of that part of the issue (I'm not certain how PCH works), each GPU will get something more like 833 MB/s theoretical maximum bandwidth.

This setup depends on you not needing to saturate the PCI-E bus, much like in GPU bitcoin mining, where a 1x connection to each of the mining GPUs was often sufficient. Putting a splitter into each of the native PCI-E slots, you get 24 GPU connections at, I think, 1x PCI-E 2.0 each, for a total transfer rate (theoretical up+down) of 1000MB/s. However, there are only 40 lanes on each CPU, so unless the PCH on the motherboard takes care of that part of the issue (I'm not certain how PCH works), each GPU will get something more like 833 MB/s theoretical maximum bandwidth.

Source Link
Adam Wykes
Adam Wykes
  • 2.4k
  • 1
  • 11
  • 20

This is a standard form-factor solution with a non-standard GPU configuration designed to overcome standard case mounting limitations and the limitations inherent to card size:

The PC itself:
PCPartPicker part list: http://pcpartpicker.com/list/TPTFtJ
Price breakdown by merchant: http://pcpartpicker.com/list/TPTFtJ/by_merchant/

CPU: Intel Xeon E5-2699 V4 2.2GHz 22-Core OEM/Tray Processor 
CPU: Intel Xeon E5-2699 V4 2.2GHz 22-Core OEM/Tray Processor 
CPU Cooler: Dynatron R27 65.4 CFM Ball Bearing CPU Cooler  ($29.88 @ OutletPC) 
CPU Cooler: Dynatron R27 65.4 CFM Ball Bearing CPU Cooler  ($29.88 @ OutletPC) 
Motherboard: Asus Z10PE-D16 WS SSI EEB Dual-CPU LGA2011-3 Motherboard  ($484.99 @ SuperBiiz) 
Memory: Kingston ValueRAM 16GB (1 x 16GB) DDR4-2133 Memory  ($73.99 @ SuperBiiz) 
Memory: Kingston ValueRAM 16GB (1 x 16GB) DDR4-2133 Memory  ($73.99 @ SuperBiiz) 
Memory: Kingston ValueRAM 16GB (1 x 16GB) DDR4-2133 Memory  ($73.99 @ SuperBiiz) 
Memory: Kingston ValueRAM 16GB (1 x 16GB) DDR4-2133 Memory  ($73.99 @ SuperBiiz) 
Storage: Western Digital Red Pro 6TB 3.5" 7200RPM Internal Hard Drive  ($279.99 @ Amazon) 
Storage: Western Digital Red Pro 6TB 3.5" 7200RPM Internal Hard Drive  ($279.99 @ Amazon) 
Storage: Western Digital Red Pro 6TB 3.5" 7200RPM Internal Hard Drive  ($279.99 @ Amazon) 
Case: Silverstone FT04B-W ATX Full Tower Case  ($229.00 @ Amazon) 
Power Supply: EVGA SuperNOVA T2 750W 80+ Titanium Certified Fully-Modular ATX Power Supply  ($167.04 @ Newegg) 
Optical Drive: Lite-On iHDS118-04 DVD/CD Drive  ($13.99 @ Newegg) 
Case Fan: Delta Electronics FFB1212EH-F00 150.3 CFM  120mm Fan  ($32.81 @ Amazon) 
Case Fan: Delta Electronics FFB1212EH-F00 150.3 CFM  120mm Fan  ($32.81 @ Amazon) 
Case Fan: Delta Electronics FFB1212EH-F00 150.3 CFM  120mm Fan  ($32.81 @ Amazon) 
Case Fan: Delta Electronics FFB1212EH-F00 150.3 CFM  120mm Fan  ($32.81 @ Amazon) 
Fan Controller: Lamptron FC-FC5V2-B Fan Controller  ($61.99 @ Amazon) 
UPS: APC BR1500G UPS  ($163.70 @ Amazon) 
Total: $2447.64
Prices include shipping, taxes, and discounts when available
Generated by PCPartPicker 2016-07-12 17:51 EDT-0400

The GPU solution:

This setup depends on you not needing to saturate the PCI-E bus, much like in GPU bitcoin mining, where a 1x connection to each of the mining GPUs was often sufficient. Putting a splitter into each of the native PCI-E slots, you get 24 GPU connections at, I think, 1x PCI-E 2.0 each, for a total transfer rate (theoretical) of 1000MB/s. However, there are only 40 lanes on each CPU, so unless the PCH on the motherboard takes care of that part of the issue (I'm not certain how PCH works), each GPU will get something more like 833 MB/s theoretical maximum bandwidth.

Blockquote (4000mbs 4x PCI-E 2.0 bandwidth / 4 gpus) * 0.8325 percent lane saturation Blockquote

There are probably other ways of doing this that would net you more CPUs and thus more lanes, and I do know they make GPU backplanes that might be of use here, but this was the configuration I was comfortable recommending and I do like that it fits nicely within the desire to keep things working on a standard format workstation motherboard.

The fans, coolers, RAM, storage, PSU are all just suggestions, not strict requirements - they are a common kind of configuration for builds like this that I have seen recommended before Note that this system, as configured, would be extremely loud and might have to run headless to make the most use of your GPU/PCI-E resources.