I'm building a multi-GPU machine for scientific calculations. I'm going to pair a Xeon E5-2620v3 CPU with the Asus X99-E WS motherboard to install 3-4 Titan X GPUs. The CPU provides 40 PCIe lanes and the chipset provides additional lanes, so that one can run up to quad x16 setups on a 40-lane CPU. What bothers me is the theoretical performance impact originating from the limited bandwidth. How critical is that for applications with almost 100% GPU-bound loads if GPUs will be constantly loading data from RAM? Should I rather go with 2 CPUs?
7
-
If it’s “constantly loading” at 1KB/s it won’t matter much. So, the answer is of course: It depends. Also, what limited bandwidth?– Daniel BCommented Oct 18, 2016 at 12:45
-
@DanielB the number of PCI-lanes (without switches) limits the data bandwidth for the GPUs. It will be constantly loading gigabytes of memory.– Eli KorvigoCommented Oct 18, 2016 at 13:00
-
I’m pretty sure the 2×8GT/s QPI is up to the job. Again, you’re not specifying the desired throughput. If I’m on ISDN, I can also be “constantly loading gigabytes” of data. You need to consider when and how the data is transferred: Constantly over time or one big transfer every once in a while? How much data is really transferred? What’s the overhead involved in transferring the data?– Daniel BCommented Oct 18, 2016 at 13:25
-
@DanielB most likely each card will have to load up to 100 GBs of data every 1-5 seconds.– Eli KorvigoCommented Oct 18, 2016 at 14:00
-
Intel ARK says: “Max Memory Bandwidth: 59 GB/s”, so that’s a no. You might want to rethink your endeavor.– Daniel BCommented Oct 18, 2016 at 14:04
|
Show 2 more comments
1 Answer
Check guides for cryptocurrency mining farms. That is pretty close to your needs. 1x PCIe throughput is enough for the most calculation-related cases. So you can wire (with rizers) up to 8 GPU per motherboard with standard windows drivers. More gpus require specialized os and drivers support.
