Assuming fairly recent Linux OS, is there an easy way to determine to which NUMA node the PCIe slot, where a device is plugged in, belongs?
5 Answers
You must go to the directory of the PCIe slot in question,
for instance eth0
:
cd /sys/class/net/eth0/device
where you will find numa_node
, local_cpus
, and local_cpulist
, the three files of interest to you. You can just cat
them, and see the desired data.
-
Huh, I got
numa_node = -1
andlocal_cpulist = 0-15
. That can't be right, I have 2 numa nodes as confirmed by hwloc.– NavinCommented Apr 15, 2016 at 22:14 -
Sorry, I have to downvote this. They asked for 'device', not 'network card'. This answer is very specific to network cards. Though it's handy and good to know, @Hem's answer, below, is easy for finding the numa node of any PCIe device on your system.– Mike SCommented Nov 1, 2022 at 1:57
-
See also: Using /sys/class to get the NUMA node for other device classes– KJ7LNWCommented Feb 26, 2023 at 5:31
The accepted answer only works for network cards, as far as I've found. Per GuillermoMA's answer, hwloc will give you the real deal even if it's not as legible. lstopo
is found in the hwloc package (at least on RHEL 7):
# lstopo
Machine (256GB)
NUMANode L#0 (P#0 128GB)
Socket L#0 + L3 L#0 (20MB)
L2 L#0 (256KB) + L1d L#0 (32KB) + L1i L#0 (32KB) + Core L#0 + PU L#0 (P#0)
L2 L#1 (256KB) + L1d L#1 (32KB) + L1i L#1 (32KB) + Core L#1 + PU L#1 (P#2)
L2 L#2 (256KB) + L1d L#2 (32KB) + L1i L#2 (32KB) + Core L#2 + PU L#2 (P#4)
L2 L#3 (256KB) + L1d L#3 (32KB) + L1i L#3 (32KB) + Core L#3 + PU L#3 (P#6)
L2 L#4 (256KB) + L1d L#4 (32KB) + L1i L#4 (32KB) + Core L#4 + PU L#4 (P#8)
L2 L#5 (256KB) + L1d L#5 (32KB) + L1i L#5 (32KB) + Core L#5 + PU L#5 (P#10)
L2 L#6 (256KB) + L1d L#6 (32KB) + L1i L#6 (32KB) + Core L#6 + PU L#6 (P#12)
L2 L#7 (256KB) + L1d L#7 (32KB) + L1i L#7 (32KB) + Core L#7 + PU L#7 (P#14)
HostBridge L#0
PCIBridge
PCI 1000:005d
Block L#0 "sda"
PCIBridge
PCI 14e4:16a1
Net L#1 "eth0"
PCI 14e4:16a1
Net L#2 "eth1"
PCI 14e4:16a1
Net L#3 "eth2"
PCI 14e4:16a1
Net L#4 "eth3"
PCI 8086:8d62
PCIBridge
PCIBridge
PCIBridge
PCIBridge
PCI 102b:0534
PCI 8086:8d02
Block L#5 "sr0"
NUMANode L#1 (P#1 128GB)
Socket L#1 + L3 L#1 (20MB)
L2 L#8 (256KB) + L1d L#8 (32KB) + L1i L#8 (32KB) + Core L#8 + PU L#8 (P#1)
L2 L#9 (256KB) + L1d L#9 (32KB) + L1i L#9 (32KB) + Core L#9 + PU L#9 (P#3)
L2 L#10 (256KB) + L1d L#10 (32KB) + L1i L#10 (32KB) + Core L#10 + PU L#10 (P#5)
L2 L#11 (256KB) + L1d L#11 (32KB) + L1i L#11 (32KB) + Core L#11 + PU L#11 (P#7)
L2 L#12 (256KB) + L1d L#12 (32KB) + L1i L#12 (32KB) + Core L#12 + PU L#12 (P#9)
L2 L#13 (256KB) + L1d L#13 (32KB) + L1i L#13 (32KB) + Core L#13 + PU L#13 (P#11)
L2 L#14 (256KB) + L1d L#14 (32KB) + L1i L#14 (32KB) + Core L#14 + PU L#14 (P#13)
L2 L#15 (256KB) + L1d L#15 (32KB) + L1i L#15 (32KB) + Core L#15 + PU L#15 (P#15)
HostBridge L#7
PCIBridge
PCI 15b3:1003
Net L#6 "eth4"
Net L#7 "eth5"
NUMANode L#0 is, of course, CPU0 and NUMANode L#1 is CPU1. You can then take your favorite PCI number from the above, eg 14e4:16a1, and discover what it is, and its PCI address for further analysis from lspci
:
# lspci -nn | grep 14e4:16a1
01:00.0 Ethernet controller [0200]: Broadcom Corporation BCM57840 NetXtreme II 10 Gigabit Ethernet [14e4:16a1] (rev 11)
01:00.1 Ethernet controller [0200]: Broadcom Corporation BCM57840 NetXtreme II 10 Gigabit Ethernet [14e4:16a1] (rev 11)
01:00.2 Ethernet controller [0200]: Broadcom Corporation BCM57840 NetXtreme II 10 Gigabit Ethernet [14e4:16a1] (rev 11)
01:00.3 Ethernet controller [0200]: Broadcom Corporation BCM57840 NetXtreme II 10 Gigabit Ethernet [14e4:16a1] (rev 11)
On one of my machines, an Emulex Fibre Channel card did not show up in the lstopo
output. I found it using lstopo --whole-io
, by performing the reverse lookup process (scroll to the right if you have to, to see the 10df hex number that I grep for):
# lspci -nn | grep -i emulex
03:00.0 Fibre Channel [0c04]: Emulex Corporation Saturn-X: LightPulse Fibre Channel Host Adapter [10df:f100] (rev 03)
03:00.1 Fibre Channel [0c04]: Emulex Corporation Saturn-X: LightPulse Fibre Channel Host Adapter [10df:f100] (rev 03)
# lstopo --whole-io | grep 10df
PCI 10df:f100
PCI 10df:f100
Remove the piped grep command, above, and rummage through the output semi-manually to find the device in the full lstopo --whole-io
display (left as an exercise for the reader).
-
lspci -nn | grep PCINUMBER
made my day. I have two Samsung 970 Pro and that command helped me to identify them from theltopo
output. Thank you.– pietropCommented Aug 10, 2018 at 7:03
You can also use hwloc (http://www.open-mpi.de/projects/hwloc/) if you know the device id. However, if you have 2 of the same device (for instance, GPUs), the only way to know the NUMA node where the physical slot is associated to is to have a look at the motherboard manual.
For the Asus Z9PE-D8 (http://dlcdnet.asus.com/pub/ASUS/mb/LGA2011/Z9PE-D8-WS/Manual/e8726_z9pe-d8_ws.pdf) it's in page 223.
-
If true, it is probably architecture dependent. For example, the the past several generations of Intel Xeon processors, the PCIe root bridge is on the CPU chip itself. Thus, knowing the CPU socket where a device's PCIe bus resides is a core requirement for operation. This is evidenced by the "/sys/class..." answer above in this case. On these same systems, two identical CPUs on different sockets would show different NUMA devices. Alternatively, on most systems, devices on different busses would have significantly different PCIe addresses, e.g. 1a:00.0 on socket 1, and 89:00.0 on socket 2.– PaulCommented Oct 16, 2017 at 22:04
First, perform dmidecode -t slot | grep -E 'Slot|Address'
. Choose your slot and associated PCIe address. For example:
# dmidecode -t slot | grep -E 'Slot|Address'
... (output edited for brevity)...
System Slot Information
Designation: PCIe Slot 6
Bus Address: 0000:18:00.0
To verify what is in your slot, perform lspci -s pci_address
. For
example, in this case our address is 18:00.0 (the leading 0000 is optional in this case):
lspci -s 18:00.0
18:00.0 RAID bus controller: Broadcom / LSI MegaRAID Tri-Mode SAS3508 (rev 01)
Now simply: # cat /sys/bus/pci/devices/<PCI device>/numa_node
(the leading 0000 is not optional in this command).
Example:
# cat /sys/bus/pci/devices/0000\:18\:00.0/numa_node
0
This device is on numa 0, as shown.
This technique works for network cards, RAID cards, Xilinx FPGA cards.
-
This is the closest to the correct answer. I'm going to edit it and make it foolproof (I hope). /sys/bus/pci/devices is easy-peasy.– Mike SCommented Nov 1, 2022 at 1:46
Someone notes that @MariusMatutiae's answer only covers net
, but that is not true: His answer trivially extends to all device classes with just a bit of globbing:
for i in /sys/class/*/*/device; do
pci=$(basename "$(readlink $i)")
if [ -e $i/numa_node ]; then
echo "NUMA Node: `cat $i/numa_node` ($i): `lspci -s $pci`" ;
fi
done | sort
Here is a sample from my system:
NUMA Node: 0 (/sys/class/drm/card0/device): c3:00.0 VGA compatible controller: ASPEED Technology, Inc. ASPEED Graphics Family (rev 41)
NUMA Node: 0 (/sys/class/net/eth0/device): c1:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
NUMA Node: 0 (/sys/class/pci_bus/0000:c1/device): c0:01.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge
NUMA Node: 0 (/sys/class/pci_bus/0000:c3/device): c2:00.0 PCI bridge: ASPEED Technology, Inc. AST1150 PCI-to-PCI Bridge (rev 04)
NUMA Node: 1 (/sys/class/nvme/nvme0/device): 84:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller SM981/PM981/PM983
NUMA Node: 1 (/sys/class/usbmon/usbmon3/device): 82:00.0 USB controller: ASMedia Technology Inc. ASM1042A USB 3.0 Host Controller
NUMA Node: 2 (/sys/class/iommu/ivhd2/device): 40:00.2 IOMMU: Advanced Micro Devices, Inc. [AMD] Starship/Matisse IOMMU
NUMA Node: 3 (/sys/class/i2c-adapter/i2c-0/device): 00:14.0 SMBus: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller (rev 61)