I know the original question is from 2019, but by the sounds of it, it might be a lot easier with more recent kernels.
For anyone having any issues, I wanted to pass through a NIC on a Broadcom BCM5709 4-port NetXtreme II Gigabit Ethernet card, which was plugged in to a HP Z600. I had already turned all virtualisation, VT-d, and the like in the BIOS.
My issue was that if I pass through one NIC on the 4-port, nothing in Group 12 was accessible to the host, including the internal NIC I was using to connect to it.
This is a useful section to give some basics on how to edit the Kernel commandline
https://pve.proxmox.com/pve-docs/chapter-sysadmin.html#sysboot_edit_kernel_cmdline
Grub
The kernel commandline needs to be placed in the variable GRUB_CMDLINE_LINUX_DEFAULT in the file /etc/default/grub. Running update-grub appends its content to all linux entries in /boot/grub/grub.cfg.
Systemd-boot
The kernel commandline needs to be placed as one line in /etc/kernel/cmdline. To apply your changes, run proxmox-boot-tool refresh [not sure what would be used without Proxmox], which sets it as the option line for all config files in loader/entries/proxmox-*.conf.
Then, this page gives some details about PCI Passthrough, primarily focussed at Proxmox, but the principle should still be the same:
https://pve.proxmox.com/wiki/PCI_Passthrough
THIS VIDEO HELPED ME THE MOST! and was really useful and thorough in walking you through the process. It is for a specific implementation, but everything shown can be accomplished using the above and below information
https://www.youtube.com/watch?v=qQiMMeVNw-o&ab_channel=SpaceinvaderOne
Finally, I can't remember where I got it from, but this is a really useful command to identify all the IOMMU Groups for all the devices (run as root):
for d in $(find /sys/kernel/iommu_groups/ -type l | sort -n -k5 -t/); do n=${d#*/iommu_groups/*}; n=${n%%/*}; printf 'IOMMU Group %s ' "$n"; lspci -nns "${d##*/}"; done;
For me, I originally started with GRUB_CMDLINE_LINUX_DEFAULT="quiet"
I enabled the basic enable IOMMU GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on"
This gave me:
IOMMU Group 12 00:1c.0 PCI bridge [0604]: Intel Corporation 82801JI (ICH10 Family) PCI Express Root Port 1 [8086:3a40]
IOMMU Group 12 00:1c.5 PCI bridge [0604]: Intel Corporation 82801JI (ICH10 Family) PCI Express Root Port 6 [8086:3a4a]
IOMMU Group 12 01:00.0 Ethernet controller [0200]: Broadcom Inc. and subsidiaries NetXtreme BCM5764M Gigabit Ethernet PCIe [14e4:1684] (rev 10)
IOMMU Group 12 1c:00.0 PCI bridge [0604]: Microsemi / PMC / IDT PES12T3G2 PCI Express Gen2 Switch [111d:8061] (rev 01)
IOMMU Group 12 1d:02.0 PCI bridge [0604]: Microsemi / PMC / IDT PES12T3G2 PCI Express Gen2 Switch [111d:8061] (rev 01)
IOMMU Group 12 1d:04.0 PCI bridge [0604]: Microsemi / PMC / IDT PES12T3G2 PCI Express Gen2 Switch [111d:8061] (rev 01)
IOMMU Group 12 1e:00.0 Ethernet controller [0200]: Broadcom Inc. and subsidiaries NetXtreme II BCM5709 Gigabit Ethernet [14e4:1639] (rev 20)
IOMMU Group 12 1e:00.1 Ethernet controller [0200]: Broadcom Inc. and subsidiaries NetXtreme II BCM5709 Gigabit Ethernet [14e4:1639] (rev 20)
IOMMU Group 12 1f:00.0 Ethernet controller [0200]: Broadcom Inc. and subsidiaries NetXtreme II BCM5709 Gigabit Ethernet [14e4:1639] (rev 20)
IOMMU Group 12 1f:00.1 Ethernet controller [0200]: Broadcom Inc. and subsidiaries NetXtreme II BCM5709 Gigabit Ethernet [14e4:1639] (rev 20)
I've removed everything except the group I'm interested in, Group 12. You can see the 4 ports of the NIC, but also another NIC, the onboard one. Unfortunately, if I pass through one NIC, nothing in Group 12 is accessible to the host.
I found a working mechanism which splits the NIC in two, using GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on pcie_acs_override=downstream"
This gave me
IOMMU Group 12 00:1c.0 PCI bridge [0604]: Intel Corporation 82801JI (ICH10 Family) PCI Express Root Port 1 [8086:3a40]
IOMMU Group 13 00:1c.5 PCI bridge [0604]: Intel Corporation 82801JI (ICH10 Family) PCI Express Root Port 6 [8086:3a4a]
IOMMU Group 18 1c:00.0 PCI bridge [0604]: Microsemi / PMC / IDT PES12T3G2 PCI Express Gen2 Switch [111d:8061] (rev 01)
IOMMU Group 19 1d:02.0 PCI bridge [0604]: Microsemi / PMC / IDT PES12T3G2 PCI Express Gen2 Switch [111d:8061] (rev 01)
IOMMU Group 20 1d:04.0 PCI bridge [0604]: Microsemi / PMC / IDT PES12T3G2 PCI Express Gen2 Switch [111d:8061] (rev 01)
IOMMU Group 21 1e:00.0 Ethernet controller [0200]: Broadcom Inc. and subsidiaries NetXtreme II BCM5709 Gigabit Ethernet [14e4:1639] (rev 20)
IOMMU Group 21 1e:00.1 Ethernet controller [0200]: Broadcom Inc. and subsidiaries NetXtreme II BCM5709 Gigabit Ethernet [14e4:1639] (rev 20)
IOMMU Group 22 1f:00.0 Ethernet controller [0200]: Broadcom Inc. and subsidiaries NetXtreme II BCM5709 Gigabit Ethernet [14e4:1639] (rev 20)
IOMMU Group 22 1f:00.1 Ethernet controller [0200]: Broadcom Inc. and subsidiaries NetXtreme II BCM5709 Gigabit Ethernet [14e4:1639] (rev 20)
IOMMU Group 23 01:00.0 Ethernet controller [0200]: Broadcom Inc. and subsidiaries NetXtreme BCM5764M Gigabit Ethernet PCIe [14e4:1684] (rev 10)
You can see that it's split the 4-port NIC into 2 pairs, and neither is in a group with the on-board NIC. I suspect each pair is on the same bus (excuse the terminology) on the card as each other, so that's probably the smallest split I could get (I did try other mechanisms, e.g. isolating one or more bridges as outlined in the video, but they were worse than the above).
Hope this might be of some use to anyone having the same issue.
