How can I understand the device topology?

Question

I am having trouble understanding the device-model and PCIe topology on my system. I've listed the output I've been looking at and attempted to explain my mental-model of what's there in hopes it helps highlight what I'm not understanding.

My output

$: lspci -t
-[0000:00]-+-00.0
           +-00.2
           +-01.0
           +-01.1-[01-03]----00.0-[02-03]----00.0-[03]--+-00.0   # 1
           |                                            \-00.1   # 2
           +-01.2-[04-0b]----00.0-[05-0b]--+-01.0-[06]----00.0
           |                               +-04.0-[07]----00.0
           |                               +-05.0-[08]----00.0
           |                               +-08.0-[09]--+-00.0
           |                               |            +-00.1
           |                               |            \-00.3
           |                               +-09.0-[0a]----00.0
           |                               \-0a.0-[0b]----00.0
# ...

$: lspci
00:00.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Renoir/Cezanne Root Complex
# ...
00:01.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Renoir PCIe Dummy Host Bridge
00:01.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Renoir PCIe GPP Bridge
# ...
01:00.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 XL Upstream Port of PCI Express Switch (rev c0)
02:00.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 XL Downstream Port of PCI Express Switch
03:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Navi 21 [Radeon RX 6800/6800 XT / 6900 XT] (rev c0)
03:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Navi 21 HDMI Audio [Radeon RX 6800/6800 XT / 6900 XT]
# ...

$ ls /sys/bus/pci/devices/
# ...
0000:00:01.0 -> ../../../devices/pci0000:00/0000:00:01.0/
0000:00:01.1 -> ../../../devices/pci0000:00/0000:00:01.1/   # 3
0000:00:01.2 -> ../../../devices/pci0000:00/0000:00:01.2/
# ...
0000:01:00.0 -> ../../../devices/pci0000:00/0000:00:01.1/0000:01:00.0/    # 4
0000:02:00.0 -> ../../../devices/pci0000:00/0000:00:01.1/0000:01:00.0/0000:02:00.0/    #5
0000:03:00.0 -> ../../../devices/pci0000:00/0000:00:01.1/0000:01:00.0/0000:02:00.0/0000:03:00.0/
0000:03:00.1 -> ../../../devices/pci0000:00/0000:00:01.1/0000:01:00.0/0000:02:00.0/0000:03:00.1/
# ...

Attempt at explaining

Device 00:00.0 is the Controller of the only bus that interfaces with the CPU, and it controls all traffic on bus 00. The CPU can interface with 00:00.0 directly in order to configure how it controls bus 00, but otherwise, it is essentially a passive tunnel for the CPU to communicate with other devices on the bus.

Device 00:01 is a physical controller on bus 00 in a "child" role (in that it operates as any other device on bus 00). It makes available 3 different functions:

• 00:01.0 Host bridge ... PCIe Dummy Host Bridge
• 00:01.1 a bridge to bus 01
• 00:01.2 a bridge to bus 02

I interpret 00:01.0 presumably as being a configuration/control type of interface for the other 2 functions. Is this a reasonable assumption?

For the bridges, from the links in /sys/bus/pci/devices, we see that 00:01.1 points to a device of the same name directly on bus 00. Also, 01:00.0 points to a device nested under 00:01.1 (lines denoted 3 and 4).

At this point, I am thinking that 01:00.0 is a bus controller (similar conceptually to 00:00.0) which needs to send all of its communication through 00:01.1 to get to the CPU.

However, 02:00.0 is further nested within 01:00.0 and there is no such similar bridge device to connect the buses. I was expecting bus 02 to either:

1. mirror the previous nesting. For example, I thought there would be 2 devices: 01:01.0 (Dummy) and 01:01.1 (bridge), and that the bridge would be nested in the directory first, and then would come bus 02.
2. or that buses 02 and 03 would be in the same directory as bus 01 e.g:

line 5:
0000:02:00.0 -> ../../../devices/pci0000:00/0000:00:01.1/0000:01:00.0/0000:02:00.0/
my expectation for what it would look like for case 1:
0000:02:00.0 -> ../../../devices/pci0000:00/0000:00:01.1/0000:01:00.0/0000:01:01.1/0000:02:00.0/
my expectation for what it would look like for case 2:
0000:02:00.0 -> ../../../devices/pci0000:00/0000:00:01.1/0000:02:00.0/

Questions

1. How can I understand the topological structure for buses 1, 2, and 3?
2. Noting that bus 3 isn't actually a bus (it is a VGA controller), how can it exist on its own bus? The Kernel driver in use for this device is amdgpu and I double checked that this is indeed a driver which registers itself as a pci driver.

Does the CPU/operating system interact with 00:00.0 only for configuration purposes

Answer 1

However, 02:00.0 is further nested within 01:00.0 and there is no such similar bridge device to connect the buses.

There is: it's the 02:00 and 01:00 device itself. As you can see in the list, both PCI IDs represent ports in a PCIe switch (i.e. they're two sides of the same device).

While classic PCI was indeed a shared bus with a mostly flat layout, PCI Express is not a shared bus anymore – it is a packet-switched network with point-to-point links, a completely different system that only imitates classic PCI behavior and inherits the terminology, but the "bus ID" no longer means exactly the same thing as it did in classic PCI.

(This mirrors various other technologies – Ethernet, SCSI, ATA – also migrating from a shared-bus system to a packet-switched system.)

So with PCI Express you have switches and point-to-point links, much like in Ethernet, and ports and bridges have "logical bus" numbers assigned in a way that makes the logical topology look like something that would make sense to a traditional PCI system. From the specification:

PCI Express maps logical PCI Bus Numbers onto PCI Express Links such that PCI 3.0 compatible configuration software views the Configuration Space of a PCI Express Hierarchy as a PCI hierarchy including multiple bus segments.

The device named Root Complex is the top level switch in the hierarchy with bus number 00, and what you see as [01-03] and [04-0b] in the tree is the range of "bus numbers" that a lower-level switch has claimed (i.e. 'Secondary Bus Number' and 'Subordinate Bus Number' registers), according to the specification.

For example, if I understand the specification correctly, 01.2-[04-0b] means that 00:01.2 is a switch/bridge that has been assigned the range of bus IDs 04–0b (hexadecimal). The first and only device connected to it gets bus ID 04 (therefore "00.0-[05-0b]" is actually 04:00.0) and is also a switch that further routes the remaining bus IDs 05–0b to its own downstream devices.

Answer 2

➜ lspci
00:00.0 Host bridge: Intel Corporation 10th Gen Core Processor Host Bridge/DRAM Registers (rev 02)
00:01.0 PCI bridge: Intel Corporation 6th-10th Gen Core Processor PCIe Controller (x16) (rev 02)
00:02.0 VGA compatible controller: Intel Corporation CometLake-H GT2 [UHD Graphics] (rev 05)
00:04.0 Signal processing controller: Intel Corporation Xeon E3-1200 v5/E3-1500 v5/6th Gen Core Processor Thermal Subsystem (rev 02)
00:08.0 System peripheral: Intel Corporation Xeon E3-1200 v5/v6 / E3-1500 v5 / 6th/7th/8th Gen Core Processor Gaussian Mixture Model
00:12.0 Signal processing controller: Intel Corporation Comet Lake PCH Thermal Controller
00:13.0 Serial controller: Intel Corporation Device 06fc
00:14.0 USB controller: Intel Corporation Comet Lake USB 3.1 xHCI Host Controller
00:14.2 RAM memory: Intel Corporation Comet Lake PCH Shared SRAM
00:14.3 Network controller: Intel Corporation Comet Lake PCH CNVi WiFi
00:15.0 Serial bus controller: Intel Corporation Comet Lake PCH Serial IO I2C Controller #0
00:15.1 Serial bus controller: Intel Corporation Comet Lake PCH Serial IO I2C Controller #1
00:16.0 Communication controller: Intel Corporation Comet Lake HECI Controller
00:1b.0 PCI bridge: Intel Corporation Comet Lake PCI Express Root Port #17 (rev f0)
00:1c.0 PCI bridge: Intel Corporation Device 06b8 (rev f0)
00:1c.4 PCI bridge: Intel Corporation Device 06bc (rev f0)
00:1d.0 PCI bridge: Intel Corporation Comet Lake PCI Express Root Port #9 (rev f0)
00:1f.0 ISA bridge: Intel Corporation Device 068e
00:1f.3 Multimedia audio controller: Intel Corporation Comet Lake PCH cAVS
00:1f.4 SMBus: Intel Corporation Comet Lake PCH SMBus Controller
00:1f.5 Serial bus controller: Intel Corporation Comet Lake PCH SPI Controller
01:00.0 VGA compatible controller: NVIDIA Corporation TU106M [GeForce RTX 2060 Max-Q] (rev a1)
01:00.1 Audio device: NVIDIA Corporation TU106 High Definition Audio Controller (rev a1)
01:00.2 USB controller: NVIDIA Corporation TU106 USB 3.1 Host Controller (rev a1)
01:00.3 Serial bus controller: NVIDIA Corporation TU106 USB Type-C UCSI Controller (rev a1)
02:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller SM981/PM981/PM983
03:00.0 Unassigned class [ff00]: Realtek Semiconductor Co., Ltd. RTS5260 PCI Express Card Reader (rev 01)
04:00.0 PCI bridge: Intel Corporation JHL7540 Thunderbolt 3 Bridge [Titan Ridge 4C 2018] (rev 06)
05:00.0 PCI bridge: Intel Corporation JHL7540 Thunderbolt 3 Bridge [Titan Ridge 4C 2018] (rev 06)
05:01.0 PCI bridge: Intel Corporation JHL7540 Thunderbolt 3 Bridge [Titan Ridge 4C 2018] (rev 06)
05:02.0 PCI bridge: Intel Corporation JHL7540 Thunderbolt 3 Bridge [Titan Ridge 4C 2018] (rev 06)
05:04.0 PCI bridge: Intel Corporation JHL7540 Thunderbolt 3 Bridge [Titan Ridge 4C 2018] (rev 06)
06:00.0 System peripheral: Intel Corporation JHL7540 Thunderbolt 3 NHI [Titan Ridge 4C 2018] (rev 06)
3a:00.0 USB controller: Intel Corporation JHL7540 Thunderbolt 3 USB Controller [Titan Ridge 4C 2018] (rev 06)
6e:00.0 PCI bridge: Intel Corporation JHL7540 Thunderbolt 3 Bridge [Titan Ridge 4C 2018] (rev 06)
6f:00.0 PCI bridge: Intel Corporation JHL7540 Thunderbolt 3 Bridge [Titan Ridge 4C 2018] (rev 06)
6f:01.0 PCI bridge: Intel Corporation JHL7540 Thunderbolt 3 Bridge [Titan Ridge 4C 2018] (rev 06)
6f:02.0 PCI bridge: Intel Corporation JHL7540 Thunderbolt 3 Bridge [Titan Ridge 4C 2018] (rev 06)
6f:04.0 PCI bridge: Intel Corporation JHL7540 Thunderbolt 3 Bridge [Titan Ridge 4C 2018] (rev 06)
70:00.0 System peripheral: Intel Corporation JHL7540 Thunderbolt 3 NHI [Titan Ridge 4C 2018] (rev 06)
a4:00.0 USB controller: Intel Corporation JHL7540 Thunderbolt 3 USB Controller [Titan Ridge 4C 2018] (rev 06)

---

➜lspci -t
-[0000:00]-+-00.0
           +-01.0-[01]--+-00.0
           |            +-00.1
           |            +-00.2
           |            \-00.3
           +-02.0
           +-04.0
           +-08.0
           +-12.0
           +-13.0
           +-14.0
           +-14.2
           +-14.3
           +-15.0
           +-15.1
           +-16.0
           +-1b.0-[02]----00.0
           +-1c.0-[03]----00.0
           +-1c.4-[04-6d]----00.0-[05-6d]--+-00.0-[06]----00.0
           |                               +-01.0-[07-39]--
           |                               +-02.0-[3a]----00.0
           |                               \-04.0-[3b-6d]--
           +-1d.0-[6e-d7]----00.0-[6f-d7]--+-00.0-[70]----00.0
           |                               +-01.0-[71-a3]--
           |                               +-02.0-[a4]----00.0
           |                               \-04.0-[a5-d7]--
           +-1f.0
           +-1f.3
           +-1f.4
           \-1f.5

PCI Express topology diagram from Wikipedia.

The domain:bus:device.function notation is used to uniquely identify each PCIe device. Usually, domain is 0000 and omitted.

01:00.0 VGA compatible controller: NVIDIA Corporation TU106M [GeForce RTX 2060 Max-Q] (rev a1)

So, you can see the discrete card here is on bus 01, device 00, function 0.

-[0000:00]-+-00.0
           +-01.0-[01]--+-00.0
           |            +-00.1
           |            +-00.2
           |            \-00.3

In the table, content in [] means bus, should concatenate with the other part to get the correspond PCIe device in the output of lspci. In the above extract, 00.0, 01.0 on the second column need to concatenate with the bus [0000:00] to get the PCIe devices, respectively, 00:00.0, 00:01.0. And 00.0, 00.1, 00.2 00.3 on the third column need to concatenate with the bus [01] to get the PCIe devices, respectively, 01:00.0, 01:00.1, .... Reasonably, bus [01] is a secondary bus created by device 00:01.0.

From the table you can see, bus 00 is the main bus (Root Complex), then 00:01.0 is a PCI bridge connected to the main bus, and it created a secondary bus 01, the first device on bus 01 has four functions, respectively 01:00.0, 01:00.1, 01:00.2, 01:00.3. You can find these devices in the output of lspci.

+-1c.4-[04-6d]----00.0-[05-6d]--+-00.0-[06]----00.0
           |                    +-01.0-[07-39]--
           |                    +-02.0-[3a]----00.0
           |                    \-04.0-[3b-6d]--

Here [04-6d] means bus 04, [05-6d] means bus 05, but it indicates the bus number 04 - 6d has been assigned to its sub buses. So the devices here should be: 00:1c.4, 04:00.0, 05:00.0, 06:00.0, 05:01.0, 05:02.0, 3a:00.0, 05:04.0. You can find all these devices in the output of `lspci'.

I actually don't understand what you have asked. But I feel if you can understand the meaning of the output of lspci -t, then you can figure out it by yourself. The table clearly shows the PCIe topology.

I didn't find any authoritative source, these are my personal educated guesses.