常用 NVIDIA 运维指令速查：互联拓扑、NUMA 与 NVLink 查询

Posted on | In , | 1.7k | 9 分钟

整理 GPU 服务器运维常用的 NVIDIA 指令：用 nvidia-smi 与 nvidia-smi topo -m 查看设备状态与互联拓扑、判断是 SYS 还是 NVLink 连接，并结合 NUMA 亲和性与 CPU 绑定排查系统规模与拓扑完整性问题。

最典型的当然有 nvidia-smi 和 nvidia-smi topo -m。前者都非常熟悉了，这里对比下两台集群的 nvidia-smi topo -m 的输出：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
	GPU0	GPU1	GPU2	GPU3	GPU4	GPU5	GPU6	GPU7	CPU Affinity	NUMA Affinity	GPU NUMA ID
GPU0	 X 	SYS	SYS	SYS	SYS	SYS	SYS	SYS	0-15,32-47	0		N/A
GPU1	SYS	 X 	SYS	SYS	SYS	SYS	SYS	SYS	0-15,32-47	0		N/A
GPU2	SYS	SYS	 X 	SYS	SYS	SYS	SYS	SYS	0-15,32-47	0		N/A
GPU3	SYS	SYS	SYS	 X 	SYS	SYS	SYS	SYS	0-15,32-47	0		N/A
GPU4	SYS	SYS	SYS	SYS	 X 	SYS	SYS	SYS	16-31,48-63	1		N/A
GPU5	SYS	SYS	SYS	SYS	SYS	 X 	SYS	SYS	16-31,48-63	1		N/A
GPU6	SYS	SYS	SYS	SYS	SYS	SYS	 X 	SYS	16-31,48-63	1		N/A
GPU7	SYS	SYS	SYS	SYS	SYS	SYS	SYS	 X 	16-31,48-63	1		N/A

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
        GPU0    GPU1    GPU2    GPU3    GPU4    GPU5    GPU6    GPU7    CPU Affinity    NUMA Affinity   GPU NUMA ID
GPU0     X      NV18    NV18    NV18    NV18    NV18    NV18    NV18    0-47,96-143     0               N/A
GPU1    NV18     X      NV18    NV18    NV18    NV18    NV18    NV18    0-47,96-143     0               N/A
GPU2    NV18    NV18     X      NV18    NV18    NV18    NV18    NV18    0-47,96-143     0               N/A
GPU3    NV18    NV18    NV18     X      NV18    NV18    NV18    NV18    0-47,96-143     0               N/A
GPU4    NV18    NV18    NV18    NV18     X      NV18    NV18    NV18    48-95,144-191   1               N/A
GPU5    NV18    NV18    NV18    NV18    NV18     X      NV18    NV18    48-95,144-191   1               N/A
GPU6    NV18    NV18    NV18    NV18    NV18    NV18     X      NV18    48-95,144-191   1               N/A
GPU7    NV18    NV18    NV18    NV18    NV18    NV18    NV18     X      48-95,144-191   1               N/A

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

通过对比这两个集群的拓扑信息，可以得出以下几个重要结论：

互联方式

  • 第一个集群：所有 GPU 之间通过 PCIe 和 NUMA（Non-Uniform Memory Access，非一致性内存访问） 节点间的 SMP 互联（标记为 SYS）

  • 第二个集群：所有 GPU 之间通过 18 条 NVLink 连接（标记为 NV18）

  • 性能影响：第二个集群的 GPU 间通信性能显著优于第一个集群，因为 NVLink 的带宽和延迟都优于 PCIe+SMP 方案

    NUMA 架构

  • 两个集群都采用双 NUMA 节点设计：

  • GPU 通信：应尽量将相关任务分配到同一 NUMA 节点内的 GPU，以避免跨 NUMA 节点的频繁数据传输

  • CPU 核心分配：

    系统规模

  • GPU 数量：两个集群都是 8 GPU 配置

  • CPU 核心总数：

    拓扑完整性

  • 每个 GPU 都与其他所有 GPU 直接相连

1
2
nvidia-smi nvlink --status -i 0
nvidia-smi nvlink --capabilities -i 0

nvlink 查询结果

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
GPU 0: NVIDIA H100 80GB HBM3 (UUID: GPU-5a10e6e5-95f7-2785-ed63-6f6147f304f7)
         Link 0: 26.562 GB/s
         Link 1: 26.562 GB/s
         Link 2: 26.562 GB/s
         Link 3: 26.562 GB/s
         Link 4: 26.562 GB/s
         Link 5: 26.562 GB/s
         Link 6: 26.562 GB/s
         Link 7: 26.562 GB/s
         Link 8: 26.562 GB/s
         Link 9: 26.562 GB/s
         Link 10: 26.562 GB/s
         Link 11: 26.562 GB/s
         Link 12: 26.562 GB/s
         Link 13: 26.562 GB/s
         Link 14: 26.562 GB/s
         Link 15: 26.562 GB/s
         Link 16: 26.562 GB/s
         Link 17: 26.562 GB/s
GPU 0: NVIDIA H100 80GB HBM3 (UUID: GPU-5a10e6e5-95f7-2785-ed63-6f6147f304f7)
         Link 0, P2P is supported: true
         Link 0, Access to system memory supported: true
         Link 0, P2P atomics supported: true
         Link 0, System memory atomics supported: true
         Link 0, SLI is supported: true
         Link 0, Link is supported: true
         Link 1, P2P is supported: true
         Link 1, Access to system memory supported: true
         Link 1, P2P atomics supported: true
         Link 1, System memory atomics supported: true
         Link 1, SLI is supported: true
         Link 1, Link is supported: true
         Link 2, P2P is supported: true
         Link 2, Access to system memory supported: true
         Link 2, P2P atomics supported: true
         Link 2, System memory atomics supported: true
         Link 2, SLI is supported: true
         Link 2, Link is supported: true
         Link 3, P2P is supported: true
         Link 3, Access to system memory supported: true
         Link 3, P2P atomics supported: true
         Link 3, System memory atomics supported: true
         Link 3, SLI is supported: true
         Link 3, Link is supported: true
         Link 4, P2P is supported: true
         Link 4, Access to system memory supported: true
         Link 4, P2P atomics supported: true
         Link 4, System memory atomics supported: true
         Link 4, SLI is supported: true
         Link 4, Link is supported: true
         Link 5, P2P is supported: true
         Link 5, Access to system memory supported: true
         Link 5, P2P atomics supported: true
         Link 5, System memory atomics supported: true
         Link 5, SLI is supported: true
         Link 5, Link is supported: true
         Link 6, P2P is supported: true
         Link 6, Access to system memory supported: true
         Link 6, P2P atomics supported: true
         Link 6, System memory atomics supported: true
         Link 6, SLI is supported: true
         Link 6, Link is supported: true
         Link 7, P2P is supported: true
         Link 7, Access to system memory supported: true
         Link 7, P2P atomics supported: true
         Link 7, System memory atomics supported: true
         Link 7, SLI is supported: true
         Link 7, Link is supported: true
         Link 8, P2P is supported: true
         Link 8, Access to system memory supported: true
         Link 8, P2P atomics supported: true
         Link 8, System memory atomics supported: true
         Link 8, SLI is supported: true
         Link 8, Link is supported: true
         Link 9, P2P is supported: true
         Link 9, Access to system memory supported: true
         Link 9, P2P atomics supported: true
         Link 9, System memory atomics supported: true
         Link 9, SLI is supported: true
         Link 9, Link is supported: true
         Link 10, P2P is supported: true
         Link 10, Access to system memory supported: true
         Link 10, P2P atomics supported: true
         Link 10, System memory atomics supported: true
         Link 10, SLI is supported: true
         Link 10, Link is supported: true
         Link 11, P2P is supported: true
         Link 11, Access to system memory supported: true
         Link 11, P2P atomics supported: true
         Link 11, System memory atomics supported: true
         Link 11, SLI is supported: true
         Link 11, Link is supported: true
         Link 12, P2P is supported: true
         Link 12, Access to system memory supported: true
         Link 12, P2P atomics supported: true
         Link 12, System memory atomics supported: true
         Link 12, SLI is supported: true
         Link 12, Link is supported: true
         Link 13, P2P is supported: true
         Link 13, Access to system memory supported: true
         Link 13, P2P atomics supported: true
         Link 13, System memory atomics supported: true
         Link 13, SLI is supported: true
         Link 13, Link is supported: true
         Link 14, P2P is supported: true
         Link 14, Access to system memory supported: true
         Link 14, P2P atomics supported: true
         Link 14, System memory atomics supported: true
         Link 14, SLI is supported: true
         Link 14, Link is supported: true
         Link 15, P2P is supported: true
         Link 15, Access to system memory supported: true
         Link 15, P2P atomics supported: true
         Link 15, System memory atomics supported: true
         Link 15, SLI is supported: true
         Link 15, Link is supported: true
         Link 16, P2P is supported: true
         Link 16, Access to system memory supported: true
         Link 16, P2P atomics supported: true
         Link 16, System memory atomics supported: true
         Link 16, SLI is supported: true
         Link 16, Link is supported: true
         Link 17, P2P is supported: true
         Link 17, Access to system memory supported: true
         Link 17, P2P atomics supported: true
         Link 17, System memory atomics supported: true
         Link 17, SLI is supported: true
         Link 17, Link is supported: true

可以分析看到一些对开发实用的特性：

  • P2P（点对点）通信
  • 系统内存访问
  • P2P原子操作
  • 系统内存原子操作
  • SLI（多GPU并行）
  • 完整的链路支持
本文结束 感谢您的阅读