Warning: Permanently added '10.8.0.3' (ED25519) to the list of known hosts. Warning: Permanently added '10.8.0.2' (ED25519) to the list of known hosts. Warning: Permanently added '10.8.0.5' (ED25519) to the list of known hosts. Warning: Permanently added '10.8.0.4' (ED25519) to the list of known hosts. Warning: Permanently added '10.8.0.6' (ED25519) to the list of known hosts. Warning: Permanently added '10.8.0.1' (ED25519) to the list of known hosts. Warning: Permanently added '10.8.0.8' (ED25519) to the list of known hosts. Warning: Permanently added '10.8.0.9' (ED25519) to the list of known hosts. #nThread 1 nGpus 1 minBytes 17179869184 maxBytes 17179869184 step: 2(factor) warmup iters: 5 iters: 20 agg iters: 1 validation: 1 graph: 0 # ============================================= 欢迎使用nccl-test集群版本 —— 大模型慢节点筛选节点工具 —— ============================================= 公司名称： 北京晶隆智算科技有限公司 系统版本： v1.0.0 发布日期： 2025年4月18日 联系人： 猜猜看 基于NCCL version开发: 2.20.5 基于CUDA Runtime Version开发: 12.4 ============================================= # Using devices # Rank 0 Group 0 Pid 3447613 on p-ef-ch-su00-gpu01-h1-aq-2f-2-cm-281-1-8u-0-1 device 0 [0000:18:00] NVIDIA H100 80GB HBM3 # Rank 1 Group 0 Pid 3447614 on p-ef-ch-su00-gpu01-h1-aq-2f-2-cm-281-1-8u-0-1 device 1 [0000:2a:00] NVIDIA H100 80GB HBM3 # Rank 2 Group 0 Pid 3447615 on p-ef-ch-su00-gpu01-h1-aq-2f-2-cm-281-1-8u-0-1 device 2 [0000:3a:00] NVIDIA H100 80GB HBM3 # Rank 3 Group 0 Pid 3447616 on p-ef-ch-su00-gpu01-h1-aq-2f-2-cm-281-1-8u-0-1 device 3 [0000:5d:00] NVIDIA H100 80GB HBM3 # Rank 4 Group 0 Pid 3447617 on p-ef-ch-su00-gpu01-h1-aq-2f-2-cm-281-1-8u-0-1 device 4 [0000:9a:00] NVIDIA H100 80GB HBM3 # Rank 5 Group 0 Pid 3447618 on p-ef-ch-su00-gpu01-h1-aq-2f-2-cm-281-1-8u-0-1 device 5 [0000:ab:00] NVIDIA H100 80GB HBM3 # Rank 6 Group 0 Pid 3447619 on p-ef-ch-su00-gpu01-h1-aq-2f-2-cm-281-1-8u-0-1 device 6 [0000:ba:00] NVIDIA H100 80GB HBM3 # Rank 7 Group 0 Pid 3447620 on p-ef-ch-su00-gpu01-h1-aq-2f-2-cm-281-1-8u-0-1 device 7 [0000:db:00] NVIDIA H100 80GB HBM3 # Rank 8 Group 0 Pid 3268280 on p-ef-ch-su00-gpu02-h1-aq-2f-2-cm-281-1-18u-0-2 device 0 [0000:18:00] NVIDIA H100 80GB HBM3 # Rank 9 Group 0 Pid 3268281 on p-ef-ch-su00-gpu02-h1-aq-2f-2-cm-281-1-18u-0-2 device 1 [0000:2a:00] NVIDIA H100 80GB HBM3 # Rank 10 Group 0 Pid 3268282 on p-ef-ch-su00-gpu02-h1-aq-2f-2-cm-281-1-18u-0-2 device 2 [0000:3a:00] NVIDIA H100 80GB HBM3 # Rank 11 Group 0 Pid 3268283 on p-ef-ch-su00-gpu02-h1-aq-2f-2-cm-281-1-18u-0-2 device 3 [0000:5d:00] NVIDIA H100 80GB HBM3 # Rank 12 Group 0 Pid 3268284 on p-ef-ch-su00-gpu02-h1-aq-2f-2-cm-281-1-18u-0-2 device 4 [0000:9a:00] NVIDIA H100 80GB HBM3 # Rank 13 Group 0 Pid 3268285 on p-ef-ch-su00-gpu02-h1-aq-2f-2-cm-281-1-18u-0-2 device 5 [0000:ab:00] NVIDIA H100 80GB HBM3 # Rank 14 Group 0 Pid 3268286 on p-ef-ch-su00-gpu02-h1-aq-2f-2-cm-281-1-18u-0-2 device 6 [0000:ba:00] NVIDIA H100 80GB HBM3 # Rank 15 Group 0 Pid 3268287 on p-ef-ch-su00-gpu02-h1-aq-2f-2-cm-281-1-18u-0-2 device 7 [0000:db:00] NVIDIA H100 80GB HBM3 # Rank 16 Group 0 Pid 3221522 on p-ef-ch-su00-gpu03-h1-aq-2f-2-cm-281-2-8u-0-3 device 0 [0000:18:00] NVIDIA H100 80GB HBM3 # Rank 17 Group 0 Pid 3221523 on p-ef-ch-su00-gpu03-h1-aq-2f-2-cm-281-2-8u-0-3 device 1 [0000:2a:00] NVIDIA H100 80GB HBM3 # Rank 18 Group 0 Pid 3221524 on p-ef-ch-su00-gpu03-h1-aq-2f-2-cm-281-2-8u-0-3 device 2 [0000:3a:00] NVIDIA H100 80GB HBM3 # Rank 19 Group 0 Pid 3221525 on p-ef-ch-su00-gpu03-h1-aq-2f-2-cm-281-2-8u-0-3 device 3 [0000:5d:00] NVIDIA H100 80GB HBM3 # Rank 20 Group 0 Pid 3221526 on p-ef-ch-su00-gpu03-h1-aq-2f-2-cm-281-2-8u-0-3 device 4 [0000:9a:00] NVIDIA H100 80GB HBM3 # Rank 21 Group 0 Pid 3221527 on p-ef-ch-su00-gpu03-h1-aq-2f-2-cm-281-2-8u-0-3 device 5 [0000:ab:00] NVIDIA H100 80GB HBM3 # Rank 22 Group 0 Pid 3221528 on p-ef-ch-su00-gpu03-h1-aq-2f-2-cm-281-2-8u-0-3 device 6 [0000:ba:00] NVIDIA H100 80GB HBM3 # Rank 23 Group 0 Pid 3221529 on p-ef-ch-su00-gpu03-h1-aq-2f-2-cm-281-2-8u-0-3 device 7 [0000:db:00] NVIDIA H100 80GB HBM3 # Rank 24 Group 0 Pid 3227088 on p-ef-ch-su00-gpu04-h1-aq-2f-2-cm-281-2-18u-0-4 device 0 [0000:18:00] NVIDIA H100 80GB HBM3 # Rank 25 Group 0 Pid 3227089 on p-ef-ch-su00-gpu04-h1-aq-2f-2-cm-281-2-18u-0-4 device 1 [0000:2a:00] NVIDIA H100 80GB HBM3 # Rank 26 Group 0 Pid 3227090 on p-ef-ch-su00-gpu04-h1-aq-2f-2-cm-281-2-18u-0-4 device 2 [0000:3a:00] NVIDIA H100 80GB HBM3 # Rank 27 Group 0 Pid 3227091 on p-ef-ch-su00-gpu04-h1-aq-2f-2-cm-281-2-18u-0-4 device 3 [0000:5d:00] NVIDIA H100 80GB HBM3 # Rank 28 Group 0 Pid 3227092 on p-ef-ch-su00-gpu04-h1-aq-2f-2-cm-281-2-18u-0-4 device 4 [0000:9a:00] NVIDIA H100 80GB HBM3 # Rank 29 Group 0 Pid 3227093 on p-ef-ch-su00-gpu04-h1-aq-2f-2-cm-281-2-18u-0-4 device 5 [0000:ab:00] NVIDIA H100 80GB HBM3 # Rank 30 Group 0 Pid 3227094 on p-ef-ch-su00-gpu04-h1-aq-2f-2-cm-281-2-18u-0-4 device 6 [0000:ba:00] NVIDIA H100 80GB HBM3 # Rank 31 Group 0 Pid 3227095 on p-ef-ch-su00-gpu04-h1-aq-2f-2-cm-281-2-18u-0-4 device 7 [0000:db:00] NVIDIA H100 80GB HBM3 # Rank 32 Group 0 Pid 3222672 on p-ef-ch-su00-gpu05-h1-aq-2f-2-cm-281-3-8u-0-5 device 0 [0000:18:00] NVIDIA H100 80GB HBM3 # Rank 33 Group 0 Pid 3222673 on p-ef-ch-su00-gpu05-h1-aq-2f-2-cm-281-3-8u-0-5 device 1 [0000:2a:00] NVIDIA H100 80GB HBM3 # Rank 34 Group 0 Pid 3222674 on p-ef-ch-su00-gpu05-h1-aq-2f-2-cm-281-3-8u-0-5 device 2 [0000:3a:00] NVIDIA H100 80GB HBM3 # Rank 35 Group 0 Pid 3222675 on p-ef-ch-su00-gpu05-h1-aq-2f-2-cm-281-3-8u-0-5 device 3 [0000:5d:00] NVIDIA H100 80GB HBM3 # Rank 36 Group 0 Pid 3222676 on p-ef-ch-su00-gpu05-h1-aq-2f-2-cm-281-3-8u-0-5 device 4 [0000:9a:00] NVIDIA H100 80GB HBM3 # Rank 37 Group 0 Pid 3222677 on p-ef-ch-su00-gpu05-h1-aq-2f-2-cm-281-3-8u-0-5 device 5 [0000:ab:00] NVIDIA H100 80GB HBM3 # Rank 38 Group 0 Pid 3222678 on p-ef-ch-su00-gpu05-h1-aq-2f-2-cm-281-3-8u-0-5 device 6 [0000:ba:00] NVIDIA H100 80GB HBM3 # Rank 39 Group 0 Pid 3222679 on p-ef-ch-su00-gpu05-h1-aq-2f-2-cm-281-3-8u-0-5 device 7 [0000:db:00] NVIDIA H100 80GB HBM3 # Rank 40 Group 0 Pid 3221371 on p-ef-ch-su00-gpu06-h1-aq-2f-2-cm-281-3-18u-0-6 device 0 [0000:18:00] NVIDIA H100 80GB HBM3 # Rank 41 Group 0 Pid 3221372 on p-ef-ch-su00-gpu06-h1-aq-2f-2-cm-281-3-18u-0-6 device 1 [0000:2a:00] NVIDIA H100 80GB HBM3 # Rank 42 Group 0 Pid 3221373 on p-ef-ch-su00-gpu06-h1-aq-2f-2-cm-281-3-18u-0-6 device 2 [0000:3a:00] NVIDIA H100 80GB HBM3 # Rank 43 Group 0 Pid 3221374 on p-ef-ch-su00-gpu06-h1-aq-2f-2-cm-281-3-18u-0-6 device 3 [0000:5d:00] NVIDIA H100 80GB HBM3 # Rank 44 Group 0 Pid 3221375 on p-ef-ch-su00-gpu06-h1-aq-2f-2-cm-281-3-18u-0-6 device 4 [0000:9a:00] NVIDIA H100 80GB HBM3 # Rank 45 Group 0 Pid 3221376 on p-ef-ch-su00-gpu06-h1-aq-2f-2-cm-281-3-18u-0-6 device 5 [0000:ab:00] NVIDIA H100 80GB HBM3 # Rank 46 Group 0 Pid 3221377 on p-ef-ch-su00-gpu06-h1-aq-2f-2-cm-281-3-18u-0-6 device 6 [0000:ba:00] NVIDIA H100 80GB HBM3 # Rank 47 Group 0 Pid 3221379 on p-ef-ch-su00-gpu06-h1-aq-2f-2-cm-281-3-18u-0-6 device 7 [0000:db:00] NVIDIA H100 80GB HBM3 # Rank 48 Group 0 Pid 3223303 on p-ef-ch-su00-gpu08-h1-aq-2f-2-cm-281-4-18u-0-8 device 0 [0000:18:00] NVIDIA H100 80GB HBM3 # Rank 49 Group 0 Pid 3223304 on p-ef-ch-su00-gpu08-h1-aq-2f-2-cm-281-4-18u-0-8 device 1 [0000:2a:00] NVIDIA H100 80GB HBM3 # Rank 50 Group 0 Pid 3223305 on p-ef-ch-su00-gpu08-h1-aq-2f-2-cm-281-4-18u-0-8 device 2 [0000:3a:00] NVIDIA H100 80GB HBM3 # Rank 51 Group 0 Pid 3223306 on p-ef-ch-su00-gpu08-h1-aq-2f-2-cm-281-4-18u-0-8 device 3 [0000:5d:00] NVIDIA H100 80GB HBM3 # Rank 52 Group 0 Pid 3223307 on p-ef-ch-su00-gpu08-h1-aq-2f-2-cm-281-4-18u-0-8 device 4 [0000:9a:00] NVIDIA H100 80GB HBM3 # Rank 53 Group 0 Pid 3223308 on p-ef-ch-su00-gpu08-h1-aq-2f-2-cm-281-4-18u-0-8 device 5 [0000:ab:00] NVIDIA H100 80GB HBM3 # Rank 54 Group 0 Pid 3223309 on p-ef-ch-su00-gpu08-h1-aq-2f-2-cm-281-4-18u-0-8 device 6 [0000:ba:00] NVIDIA H100 80GB HBM3 # Rank 55 Group 0 Pid 3223310 on p-ef-ch-su00-gpu08-h1-aq-2f-2-cm-281-4-18u-0-8 device 7 [0000:db:00] NVIDIA H100 80GB HBM3 # Rank 56 Group 0 Pid 3223459 on p-ef-ch-su00-gpu09-h1-aq-2f-2-cm-281-5-8u-0-9 device 0 [0000:18:00] NVIDIA H100 80GB HBM3 # Rank 57 Group 0 Pid 3223460 on p-ef-ch-su00-gpu09-h1-aq-2f-2-cm-281-5-8u-0-9 device 1 [0000:2a:00] NVIDIA H100 80GB HBM3 # Rank 58 Group 0 Pid 3223461 on p-ef-ch-su00-gpu09-h1-aq-2f-2-cm-281-5-8u-0-9 device 2 [0000:3a:00] NVIDIA H100 80GB HBM3 # Rank 59 Group 0 Pid 3223462 on p-ef-ch-su00-gpu09-h1-aq-2f-2-cm-281-5-8u-0-9 device 3 [0000:5d:00] NVIDIA H100 80GB HBM3 # Rank 60 Group 0 Pid 3223463 on p-ef-ch-su00-gpu09-h1-aq-2f-2-cm-281-5-8u-0-9 device 4 [0000:9a:00] NVIDIA H100 80GB HBM3 # Rank 61 Group 0 Pid 3223464 on p-ef-ch-su00-gpu09-h1-aq-2f-2-cm-281-5-8u-0-9 device 5 [0000:ab:00] NVIDIA H100 80GB HBM3 # Rank 62 Group 0 Pid 3223465 on p-ef-ch-su00-gpu09-h1-aq-2f-2-cm-281-5-8u-0-9 device 6 [0000:ba:00] NVIDIA H100 80GB HBM3 # Rank 63 Group 0 Pid 3223466 on p-ef-ch-su00-gpu09-h1-aq-2f-2-cm-281-5-8u-0-9 device 7 [0000:db:00] NVIDIA H100 80GB HBM3 NCCL version 2.20.5+cuda12.4 # # out-of-place in-place # size count type redop root time algbw busbw #wrong time algbw busbw #wrong # (B) (elements) (us) (GB/s) (GB/s) (us) (GB/s) (GB/s) 17179869184 4294967296 float sum -1 87055 197.35 388.52 0 86789 197.95 389.71 0 # Out of bounds values : 0 OK # Avg bus bandwidth : 389.118 # 参与的服务器数量=8,总卡数=64 # Warning: Permanently added '10.8.0.3' (ED25519) to the list of known hosts. Warning: Permanently added '10.8.0.6' (ED25519) to the list of known hosts. Warning: Permanently added '10.8.0.1' (ED25519) to the list of known hosts. Warning: Permanently added '10.8.0.2' (ED25519) to the list of known hosts. Warning: Permanently added '10.8.0.10' (ED25519) to the list of known hosts. Warning: Permanently added '10.8.0.8' (ED25519) to the list of known hosts. Warning: Permanently added '10.8.0.9' (ED25519) to the list of known hosts. Warning: Permanently added '10.8.0.13' (ED25519) to the list of known hosts. Warning: Permanently added '10.8.0.17' (ED25519) to the list of known hosts. Warning: Permanently added '10.8.0.15' (ED25519) to the list of known hosts. Warning: Permanently added '10.8.0.12' (ED25519) to the list of known hosts. Warning: Permanently added '10.8.0.11' (ED25519) to the list of known hosts. Warning: Permanently added '10.8.0.5' (ED25519) to the list of known hosts. Warning: Permanently added '10.8.0.14' (ED25519) to the list of known hosts. Warning: Permanently added '10.8.0.16' (ED25519) to the list of known hosts. Warning: Permanently added '10.8.0.4' (ED25519) to the list of known hosts. #nThread 1 nGpus 1 minBytes 17179869184 maxBytes 17179869184 step: 2(factor) warmup iters: 5 iters: 20 agg iters: 1 validation: 1 graph: 0 # ============================================= 欢迎使用nccl-test集群版本 —— 大模型慢节点筛选节点工具 —— ============================================= 公司名称： 北京晶隆智算科技有限公司 系统版本： v1.0.0 发布日期： 2025年4月18日 联系人： 猜猜看 基于NCCL version开发: 2.20.5 基于CUDA Runtime Version开发: 12.4 ============================================= # Using devices # Rank 0 Group 0 Pid 3448444 on p-ef-ch-su00-gpu01-h1-aq-2f-2-cm-281-1-8u-0-1 device 0 [0000:18:00] NVIDIA H100 80GB HBM3 # Rank 1 Group 0 Pid 3448445 on p-ef-ch-su00-gpu01-h1-aq-2f-2-cm-281-1-8u-0-1 device 1 [0000:2a:00] NVIDIA H100 80GB HBM3 # Rank 2 Group 0 Pid 3448446 on p-ef-ch-su00-gpu01-h1-aq-2f-2-cm-281-1-8u-0-1 device 2 [0000:3a:00] NVIDIA H100 80GB HBM3 # Rank 3 Group 0 Pid 3448447 on p-ef-ch-su00-gpu01-h1-aq-2f-2-cm-281-1-8u-0-1 device 3 [0000:5d:00] NVIDIA H100 80GB HBM3 # Rank 4 Group 0 Pid 3448448 on p-ef-ch-su00-gpu01-h1-aq-2f-2-cm-281-1-8u-0-1 device 4 [0000:9a:00] NVIDIA H100 80GB HBM3 # Rank 5 Group 0 Pid 3448449 on p-ef-ch-su00-gpu01-h1-aq-2f-2-cm-281-1-8u-0-1 device 5 [0000:ab:00] NVIDIA H100 80GB HBM3 # Rank 6 Group 0 Pid 3448450 on p-ef-ch-su00-gpu01-h1-aq-2f-2-cm-281-1-8u-0-1 device 6 [0000:ba:00] NVIDIA H100 80GB HBM3 # Rank 7 Group 0 Pid 3448451 on p-ef-ch-su00-gpu01-h1-aq-2f-2-cm-281-1-8u-0-1 device 7 [0000:db:00] NVIDIA H100 80GB HBM3 # Rank 8 Group 0 Pid 3269107 on p-ef-ch-su00-gpu02-h1-aq-2f-2-cm-281-1-18u-0-2 device 0 [0000:18:00] NVIDIA H100 80GB HBM3 # Rank 9 Group 0 Pid 3269108 on p-ef-ch-su00-gpu02-h1-aq-2f-2-cm-281-1-18u-0-2 device 1 [0000:2a:00] NVIDIA H100 80GB HBM3 # Rank 10 Group 0 Pid 3269109 on p-ef-ch-su00-gpu02-h1-aq-2f-2-cm-281-1-18u-0-2 device 2 [0000:3a:00] NVIDIA H100 80GB HBM3 # Rank 11 Group 0 Pid 3269110 on p-ef-ch-su00-gpu02-h1-aq-2f-2-cm-281-1-18u-0-2 device 3 [0000:5d:00] NVIDIA H100 80GB HBM3 # Rank 12 Group 0 Pid 3269111 on p-ef-ch-su00-gpu02-h1-aq-2f-2-cm-281-1-18u-0-2 device 4 [0000:9a:00] NVIDIA H100 80GB HBM3 # Rank 13 Group 0 Pid 3269112 on p-ef-ch-su00-gpu02-h1-aq-2f-2-cm-281-1-18u-0-2 device 5 [0000:ab:00] NVIDIA H100 80GB HBM3 # Rank 14 Group 0 Pid 3269113 on p-ef-ch-su00-gpu02-h1-aq-2f-2-cm-281-1-18u-0-2 device 6 [0000:ba:00] NVIDIA H100 80GB HBM3 # Rank 15 Group 0 Pid 3269114 on p-ef-ch-su00-gpu02-h1-aq-2f-2-cm-281-1-18u-0-2 device 7 [0000:db:00] NVIDIA H100 80GB HBM3 # Rank 16 Group 0 Pid 3222330 on p-ef-ch-su00-gpu03-h1-aq-2f-2-cm-281-2-8u-0-3 device 0 [0000:18:00] NVIDIA H100 80GB HBM3 # Rank 17 Group 0 Pid 3222331 on p-ef-ch-su00-gpu03-h1-aq-2f-2-cm-281-2-8u-0-3 device 1 [0000:2a:00] NVIDIA H100 80GB HBM3 # Rank 18 Group 0 Pid 3222332 on p-ef-ch-su00-gpu03-h1-aq-2f-2-cm-281-2-8u-0-3 device 2 [0000:3a:00] NVIDIA H100 80GB HBM3 # Rank 19 Group 0 Pid 3222333 on p-ef-ch-su00-gpu03-h1-aq-2f-2-cm-281-2-8u-0-3 device 3 [0000:5d:00] NVIDIA H100 80GB HBM3 # Rank 20 Group 0 Pid 3222334 on p-ef-ch-su00-gpu03-h1-aq-2f-2-cm-281-2-8u-0-3 device 4 [0000:9a:00] NVIDIA H100 80GB HBM3 # Rank 21 Group 0 Pid 3222335 on p-ef-ch-su00-gpu03-h1-aq-2f-2-cm-281-2-8u-0-3 device 5 [0000:ab:00] NVIDIA H100 80GB HBM3 # Rank 22 Group 0 Pid 3222336 on p-ef-ch-su00-gpu03-h1-aq-2f-2-cm-281-2-8u-0-3 device 6 [0000:ba:00] NVIDIA H100 80GB HBM3 # Rank 23 Group 0 Pid 3222337 on p-ef-ch-su00-gpu03-h1-aq-2f-2-cm-281-2-8u-0-3 device 7 [0000:db:00] NVIDIA H100 80GB HBM3 # Rank 24 Group 0 Pid 3227913 on p-ef-ch-su00-gpu04-h1-aq-2f-2-cm-281-2-18u-0-4 device 0 [0000:18:00] NVIDIA H100 80GB HBM3 # Rank 25 Group 0 Pid 3227914 on p-ef-ch-su00-gpu04-h1-aq-2f-2-cm-281-2-18u-0-4 device 1 [0000:2a:00] NVIDIA H100 80GB HBM3 # Rank 26 Group 0 Pid 3227915 on p-ef-ch-su00-gpu04-h1-aq-2f-2-cm-281-2-18u-0-4 device 2 [0000:3a:00] NVIDIA H100 80GB HBM3 # Rank 27 Group 0 Pid 3227916 on p-ef-ch-su00-gpu04-h1-aq-2f-2-cm-281-2-18u-0-4 device 3 [0000:5d:00] NVIDIA H100 80GB HBM3 # Rank 28 Group 0 Pid 3227917 on p-ef-ch-su00-gpu04-h1-aq-2f-2-cm-281-2-18u-0-4 device 4 [0000:9a:00] NVIDIA H100 80GB HBM3 # Rank 29 Group 0 Pid 3227918 on p-ef-ch-su00-gpu04-h1-aq-2f-2-cm-281-2-18u-0-4 device 5 [0000:ab:00] NVIDIA H100 80GB HBM3 # Rank 30 Group 0 Pid 3227919 on p-ef-ch-su00-gpu04-h1-aq-2f-2-cm-281-2-18u-0-4 device 6 [0000:ba:00] NVIDIA H100 80GB HBM3 # Rank 31 Group 0 Pid 3227920 on p-ef-ch-su00-gpu04-h1-aq-2f-2-cm-281-2-18u-0-4 device 7 [0000:db:00] NVIDIA H100 80GB HBM3 # Rank 32 Group 0 Pid 3223481 on p-ef-ch-su00-gpu05-h1-aq-2f-2-cm-281-3-8u-0-5 device 0 [0000:18:00] NVIDIA H100 80GB HBM3 # Rank 33 Group 0 Pid 3223482 on p-ef-ch-su00-gpu05-h1-aq-2f-2-cm-281-3-8u-0-5 device 1 [0000:2a:00] NVIDIA H100 80GB HBM3 # Rank 34 Group 0 Pid 3223483 on p-ef-ch-su00-gpu05-h1-aq-2f-2-cm-281-3-8u-0-5 device 2 [0000:3a:00] NVIDIA H100 80GB HBM3 # Rank 35 Group 0 Pid 3223484 on p-ef-ch-su00-gpu05-h1-aq-2f-2-cm-281-3-8u-0-5 device 3 [0000:5d:00] NVIDIA H100 80GB HBM3 # Rank 36 Group 0 Pid 3223485 on p-ef-ch-su00-gpu05-h1-aq-2f-2-cm-281-3-8u-0-5 device 4 [0000:9a:00] NVIDIA H100 80GB HBM3 # Rank 37 Group 0 Pid 3223486 on p-ef-ch-su00-gpu05-h1-aq-2f-2-cm-281-3-8u-0-5 device 5 [0000:ab:00] NVIDIA H100 80GB HBM3 # Rank 38 Group 0 Pid 3223487 on p-ef-ch-su00-gpu05-h1-aq-2f-2-cm-281-3-8u-0-5 device 6 [0000:ba:00] NVIDIA H100 80GB HBM3 # Rank 39 Group 0 Pid 3223488 on p-ef-ch-su00-gpu05-h1-aq-2f-2-cm-281-3-8u-0-5 device 7 [0000:db:00] NVIDIA H100 80GB HBM3 # Rank 40 Group 0 Pid 3222188 on p-ef-ch-su00-gpu06-h1-aq-2f-2-cm-281-3-18u-0-6 device 0 [0000:18:00] NVIDIA H100 80GB HBM3 # Rank 41 Group 0 Pid 3222189 on p-ef-ch-su00-gpu06-h1-aq-2f-2-cm-281-3-18u-0-6 device 1 [0000:2a:00] NVIDIA H100 80GB HBM3 # Rank 42 Group 0 Pid 3222190 on p-ef-ch-su00-gpu06-h1-aq-2f-2-cm-281-3-18u-0-6 device 2 [0000:3a:00] NVIDIA H100 80GB HBM3 # Rank 43 Group 0 Pid 3222191 on p-ef-ch-su00-gpu06-h1-aq-2f-2-cm-281-3-18u-0-6 device 3 [0000:5d:00] NVIDIA H100 80GB HBM3 # Rank 44 Group 0 Pid 3222192 on p-ef-ch-su00-gpu06-h1-aq-2f-2-cm-281-3-18u-0-6 device 4 [0000:9a:00] NVIDIA H100 80GB HBM3 # Rank 45 Group 0 Pid 3222193 on p-ef-ch-su00-gpu06-h1-aq-2f-2-cm-281-3-18u-0-6 device 5 [0000:ab:00] NVIDIA H100 80GB HBM3 # Rank 46 Group 0 Pid 3222194 on p-ef-ch-su00-gpu06-h1-aq-2f-2-cm-281-3-18u-0-6 device 6 [0000:ba:00] NVIDIA H100 80GB HBM3 # Rank 47 Group 0 Pid 3222195 on p-ef-ch-su00-gpu06-h1-aq-2f-2-cm-281-3-18u-0-6 device 7 [0000:db:00] NVIDIA H100 80GB HBM3 # Rank 48 Group 0 Pid 3224111 on p-ef-ch-su00-gpu08-h1-aq-2f-2-cm-281-4-18u-0-8 device 0 [0000:18:00] NVIDIA H100 80GB HBM3 # Rank 49 Group 0 Pid 3224112 on p-ef-ch-su00-gpu08-h1-aq-2f-2-cm-281-4-18u-0-8 device 1 [0000:2a:00] NVIDIA H100 80GB HBM3 # Rank 50 Group 0 Pid 3224113 on p-ef-ch-su00-gpu08-h1-aq-2f-2-cm-281-4-18u-0-8 device 2 [0000:3a:00] NVIDIA H100 80GB HBM3 # Rank 51 Group 0 Pid 3224114 on p-ef-ch-su00-gpu08-h1-aq-2f-2-cm-281-4-18u-0-8 device 3 [0000:5d:00] NVIDIA H100 80GB HBM3 # Rank 52 Group 0 Pid 3224115 on p-ef-ch-su00-gpu08-h1-aq-2f-2-cm-281-4-18u-0-8 device 4 [0000:9a:00] NVIDIA H100 80GB HBM3 # Rank 53 Group 0 Pid 3224116 on p-ef-ch-su00-gpu08-h1-aq-2f-2-cm-281-4-18u-0-8 device 5 [0000:ab:00] NVIDIA H100 80GB HBM3 # Rank 54 Group 0 Pid 3224117 on p-ef-ch-su00-gpu08-h1-aq-2f-2-cm-281-4-18u-0-8 device 6 [0000:ba:00] NVIDIA H100 80GB HBM3 # Rank 55 Group 0 Pid 3224118 on p-ef-ch-su00-gpu08-h1-aq-2f-2-cm-281-4-18u-0-8 device 7 [0000:db:00] NVIDIA H100 80GB HBM3 # Rank 56 Group 0 Pid 3224270 on p-ef-ch-su00-gpu09-h1-aq-2f-2-cm-281-5-8u-0-9 device 0 [0000:18:00] NVIDIA H100 80GB HBM3 # Rank 57 Group 0 Pid 3224271 on p-ef-ch-su00-gpu09-h1-aq-2f-2-cm-281-5-8u-0-9 device 1 [0000:2a:00] NVIDIA H100 80GB HBM3 # Rank 58 Group 0 Pid 3224272 on p-ef-ch-su00-gpu09-h1-aq-2f-2-cm-281-5-8u-0-9 device 2 [0000:3a:00] NVIDIA H100 80GB HBM3 # Rank 59 Group 0 Pid 3224273 on p-ef-ch-su00-gpu09-h1-aq-2f-2-cm-281-5-8u-0-9 device 3 [0000:5d:00] NVIDIA H100 80GB HBM3 # Rank 60 Group 0 Pid 3224274 on p-ef-ch-su00-gpu09-h1-aq-2f-2-cm-281-5-8u-0-9 device 4 [0000:9a:00] NVIDIA H100 80GB HBM3 # Rank 61 Group 0 Pid 3224275 on p-ef-ch-su00-gpu09-h1-aq-2f-2-cm-281-5-8u-0-9 device 5 [0000:ab:00] NVIDIA H100 80GB HBM3 # Rank 62 Group 0 Pid 3224276 on p-ef-ch-su00-gpu09-h1-aq-2f-2-cm-281-5-8u-0-9 device 6 [0000:ba:00] NVIDIA H100 80GB HBM3 # Rank 63 Group 0 Pid 3224277 on p-ef-ch-su00-gpu09-h1-aq-2f-2-cm-281-5-8u-0-9 device 7 [0000:db:00] NVIDIA H100 80GB HBM3 # Rank 64 Group 0 Pid 3216505 on p-ef-ch-su00-gpu10-h1-aq-2f-2-cm-281-5-18u-0-10 device 0 [0000:18:00] NVIDIA H100 80GB HBM3 # Rank 65 Group 0 Pid 3216506 on p-ef-ch-su00-gpu10-h1-aq-2f-2-cm-281-5-18u-0-10 device 1 [0000:2a:00] NVIDIA H100 80GB HBM3 # Rank 66 Group 0 Pid 3216507 on p-ef-ch-su00-gpu10-h1-aq-2f-2-cm-281-5-18u-0-10 device 2 [0000:3a:00] NVIDIA H100 80GB HBM3 # Rank 67 Group 0 Pid 3216508 on p-ef-ch-su00-gpu10-h1-aq-2f-2-cm-281-5-18u-0-10 device 3 [0000:5d:00] NVIDIA H100 80GB HBM3 # Rank 68 Group 0 Pid 3216509 on p-ef-ch-su00-gpu10-h1-aq-2f-2-cm-281-5-18u-0-10 device 4 [0000:9a:00] NVIDIA H100 80GB HBM3 # Rank 69 Group 0 Pid 3216510 on p-ef-ch-su00-gpu10-h1-aq-2f-2-cm-281-5-18u-0-10 device 5 [0000:ab:00] NVIDIA H100 80GB HBM3 # Rank 70 Group 0 Pid 3216511 on p-ef-ch-su00-gpu10-h1-aq-2f-2-cm-281-5-18u-0-10 device 6 [0000:ba:00] NVIDIA H100 80GB HBM3 # Rank 71 Group 0 Pid 3216512 on p-ef-ch-su00-gpu10-h1-aq-2f-2-cm-281-5-18u-0-10 device 7 [0000:db:00] NVIDIA H100 80GB HBM3 # Rank 72 Group 0 Pid 2124359 on p-ef-ch-su00-gpu11-h1-aq-2f-2-cm-281-6-8u-0-11 device 0 [0000:18:00] NVIDIA H100 80GB HBM3 # Rank 73 Group 0 Pid 2124360 on p-ef-ch-su00-gpu11-h1-aq-2f-2-cm-281-6-8u-0-11 device 1 [0000:2a:00] NVIDIA H100 80GB HBM3 # Rank 74 Group 0 Pid 2124361 on p-ef-ch-su00-gpu11-h1-aq-2f-2-cm-281-6-8u-0-11 device 2 [0000:3a:00] NVIDIA H100 80GB HBM3 # Rank 75 Group 0 Pid 2124362 on p-ef-ch-su00-gpu11-h1-aq-2f-2-cm-281-6-8u-0-11 device 3 [0000:5d:00] NVIDIA H100 80GB HBM3 # Rank 76 Group 0 Pid 2124363 on p-ef-ch-su00-gpu11-h1-aq-2f-2-cm-281-6-8u-0-11 device 4 [0000:9a:00] NVIDIA H100 80GB HBM3 # Rank 77 Group 0 Pid 2124364 on p-ef-ch-su00-gpu11-h1-aq-2f-2-cm-281-6-8u-0-11 device 5 [0000:ab:00] NVIDIA H100 80GB HBM3 # Rank 78 Group 0 Pid 2124365 on p-ef-ch-su00-gpu11-h1-aq-2f-2-cm-281-6-8u-0-11 device 6 [0000:ba:00] NVIDIA H100 80GB HBM3 # Rank 79 Group 0 Pid 2124366 on p-ef-ch-su00-gpu11-h1-aq-2f-2-cm-281-6-8u-0-11 device 7 [0000:db:00] NVIDIA H100 80GB HBM3 # Rank 80 Group 0 Pid 2123886 on p-ef-ch-su00-gpu12-h1-aq-2f-2-cm-281-6-18u-0-12 device 0 [0000:18:00] NVIDIA H100 80GB HBM3 # Rank 81 Group 0 Pid 2123887 on p-ef-ch-su00-gpu12-h1-aq-2f-2-cm-281-6-18u-0-12 device 1 [0000:2a:00] NVIDIA H100 80GB HBM3 # Rank 82 Group 0 Pid 2123888 on p-ef-ch-su00-gpu12-h1-aq-2f-2-cm-281-6-18u-0-12 device 2 [0000:3a:00] NVIDIA H100 80GB HBM3 # Rank 83 Group 0 Pid 2123889 on p-ef-ch-su00-gpu12-h1-aq-2f-2-cm-281-6-18u-0-12 device 3 [0000:5d:00] NVIDIA H100 80GB HBM3 # Rank 84 Group 0 Pid 2123890 on p-ef-ch-su00-gpu12-h1-aq-2f-2-cm-281-6-18u-0-12 device 4 [0000:9a:00] NVIDIA H100 80GB HBM3 # Rank 85 Group 0 Pid 2123891 on p-ef-ch-su00-gpu12-h1-aq-2f-2-cm-281-6-18u-0-12 device 5 [0000:ab:00] NVIDIA H100 80GB HBM3 # Rank 86 Group 0 Pid 2123892 on p-ef-ch-su00-gpu12-h1-aq-2f-2-cm-281-6-18u-0-12 device 6 [0000:ba:00] NVIDIA H100 80GB HBM3 # Rank 87 Group 0 Pid 2123893 on p-ef-ch-su00-gpu12-h1-aq-2f-2-cm-281-6-18u-0-12 device 7 [0000:db:00] NVIDIA H100 80GB HBM3 # Rank 88 Group 0 Pid 3219308 on p-ef-ch-su00-gpu13-h1-aq-2f-2-cm-281-7-8u-0-13 device 0 [0000:18:00] NVIDIA H100 80GB HBM3 # Rank 89 Group 0 Pid 3219309 on p-ef-ch-su00-gpu13-h1-aq-2f-2-cm-281-7-8u-0-13 device 1 [0000:2a:00] NVIDIA H100 80GB HBM3 # Rank 90 Group 0 Pid 3219310 on p-ef-ch-su00-gpu13-h1-aq-2f-2-cm-281-7-8u-0-13 device 2 [0000:3a:00] NVIDIA H100 80GB HBM3 # Rank 91 Group 0 Pid 3219311 on p-ef-ch-su00-gpu13-h1-aq-2f-2-cm-281-7-8u-0-13 device 3 [0000:5d:00] NVIDIA H100 80GB HBM3 # Rank 92 Group 0 Pid 3219312 on p-ef-ch-su00-gpu13-h1-aq-2f-2-cm-281-7-8u-0-13 device 4 [0000:9a:00] NVIDIA H100 80GB HBM3 # Rank 93 Group 0 Pid 3219313 on p-ef-ch-su00-gpu13-h1-aq-2f-2-cm-281-7-8u-0-13 device 5 [0000:ab:00] NVIDIA H100 80GB HBM3 # Rank 94 Group 0 Pid 3219314 on p-ef-ch-su00-gpu13-h1-aq-2f-2-cm-281-7-8u-0-13 device 6 [0000:ba:00] NVIDIA H100 80GB HBM3 # Rank 95 Group 0 Pid 3219315 on p-ef-ch-su00-gpu13-h1-aq-2f-2-cm-281-7-8u-0-13 device 7 [0000:db:00] NVIDIA H100 80GB HBM3 # Rank 96 Group 0 Pid 3219021 on p-ef-ch-su00-gpu14-h1-aq-2f-2-cm-281-7-18u-0-14 device 0 [0000:18:00] NVIDIA H100 80GB HBM3 # Rank 97 Group 0 Pid 3219022 on p-ef-ch-su00-gpu14-h1-aq-2f-2-cm-281-7-18u-0-14 device 1 [0000:2a:00] NVIDIA H100 80GB HBM3 # Rank 98 Group 0 Pid 3219023 on p-ef-ch-su00-gpu14-h1-aq-2f-2-cm-281-7-18u-0-14 device 2 [0000:3a:00] NVIDIA H100 80GB HBM3 # Rank 99 Group 0 Pid 3219024 on p-ef-ch-su00-gpu14-h1-aq-2f-2-cm-281-7-18u-0-14 device 3 [0000:5d:00] NVIDIA H100 80GB HBM3 # Rank 100 Group 0 Pid 3219025 on p-ef-ch-su00-gpu14-h1-aq-2f-2-cm-281-7-18u-0-14 device 4 [0000:9a:00] NVIDIA H100 80GB HBM3 # Rank 101 Group 0 Pid 3219026 on p-ef-ch-su00-gpu14-h1-aq-2f-2-cm-281-7-18u-0-14 device 5 [0000:ab:00] NVIDIA H100 80GB HBM3 # Rank 102 Group 0 Pid 3219027 on p-ef-ch-su00-gpu14-h1-aq-2f-2-cm-281-7-18u-0-14 device 6 [0000:ba:00] NVIDIA H100 80GB HBM3 # Rank 103 Group 0 Pid 3219028 on p-ef-ch-su00-gpu14-h1-aq-2f-2-cm-281-7-18u-0-14 device 7 [0000:db:00] NVIDIA H100 80GB HBM3 # Rank 104 Group 0 Pid 2745225 on p-ef-ch-su00-gpu15-h1-aq-2f-2-cm-281-8-8u-0-15 device 0 [0000:18:00] NVIDIA H100 80GB HBM3 # Rank 105 Group 0 Pid 2745226 on p-ef-ch-su00-gpu15-h1-aq-2f-2-cm-281-8-8u-0-15 device 1 [0000:2a:00] NVIDIA H100 80GB HBM3 # Rank 106 Group 0 Pid 2745227 on p-ef-ch-su00-gpu15-h1-aq-2f-2-cm-281-8-8u-0-15 device 2 [0000:3a:00] NVIDIA H100 80GB HBM3 # Rank 107 Group 0 Pid 2745228 on p-ef-ch-su00-gpu15-h1-aq-2f-2-cm-281-8-8u-0-15 device 3 [0000:5d:00] NVIDIA H100 80GB HBM3 # Rank 108 Group 0 Pid 2745229 on p-ef-ch-su00-gpu15-h1-aq-2f-2-cm-281-8-8u-0-15 device 4 [0000:9a:00] NVIDIA H100 80GB HBM3 # Rank 109 Group 0 Pid 2745230 on p-ef-ch-su00-gpu15-h1-aq-2f-2-cm-281-8-8u-0-15 device 5 [0000:ab:00] NVIDIA H100 80GB HBM3 # Rank 110 Group 0 Pid 2745231 on p-ef-ch-su00-gpu15-h1-aq-2f-2-cm-281-8-8u-0-15 device 6 [0000:ba:00] NVIDIA H100 80GB HBM3 # Rank 111 Group 0 Pid 2745232 on p-ef-ch-su00-gpu15-h1-aq-2f-2-cm-281-8-8u-0-15 device 7 [0000:db:00] NVIDIA H100 80GB HBM3 # Rank 112 Group 0 Pid 3215887 on p-ef-ch-su00-gpu16-h1-aq-2f-2-cm-281-8-18u-0-16 device 0 [0000:18:00] NVIDIA H100 80GB HBM3 # Rank 113 Group 0 Pid 3215888 on p-ef-ch-su00-gpu16-h1-aq-2f-2-cm-281-8-18u-0-16 device 1 [0000:2a:00] NVIDIA H100 80GB HBM3 # Rank 114 Group 0 Pid 3215889 on p-ef-ch-su00-gpu16-h1-aq-2f-2-cm-281-8-18u-0-16 device 2 [0000:3a:00] NVIDIA H100 80GB HBM3 # Rank 115 Group 0 Pid 3215890 on p-ef-ch-su00-gpu16-h1-aq-2f-2-cm-281-8-18u-0-16 device 3 [0000:5d:00] NVIDIA H100 80GB HBM3 # Rank 116 Group 0 Pid 3215891 on p-ef-ch-su00-gpu16-h1-aq-2f-2-cm-281-8-18u-0-16 device 4 [0000:9a:00] NVIDIA H100 80GB HBM3 # Rank 117 Group 0 Pid 3215892 on p-ef-ch-su00-gpu16-h1-aq-2f-2-cm-281-8-18u-0-16 device 5 [0000:ab:00] NVIDIA H100 80GB HBM3 # Rank 118 Group 0 Pid 3215893 on p-ef-ch-su00-gpu16-h1-aq-2f-2-cm-281-8-18u-0-16 device 6 [0000:ba:00] NVIDIA H100 80GB HBM3 # Rank 119 Group 0 Pid 3215894 on p-ef-ch-su00-gpu16-h1-aq-2f-2-cm-281-8-18u-0-16 device 7 [0000:db:00] NVIDIA H100 80GB HBM3 # Rank 120 Group 0 Pid 3218583 on p-ef-ch-su00-gpu17-h1-aq-2f-2-cm-281-9-8u-0-17 device 0 [0000:18:00] NVIDIA H100 80GB HBM3 # Rank 121 Group 0 Pid 3218584 on p-ef-ch-su00-gpu17-h1-aq-2f-2-cm-281-9-8u-0-17 device 1 [0000:2a:00] NVIDIA H100 80GB HBM3 # Rank 122 Group 0 Pid 3218585 on p-ef-ch-su00-gpu17-h1-aq-2f-2-cm-281-9-8u-0-17 device 2 [0000:3a:00] NVIDIA H100 80GB HBM3 # Rank 123 Group 0 Pid 3218586 on p-ef-ch-su00-gpu17-h1-aq-2f-2-cm-281-9-8u-0-17 device 3 [0000:5d:00] NVIDIA H100 80GB HBM3 # Rank 124 Group 0 Pid 3218587 on p-ef-ch-su00-gpu17-h1-aq-2f-2-cm-281-9-8u-0-17 device 4 [0000:9a:00] NVIDIA H100 80GB HBM3 # Rank 125 Group 0 Pid 3218588 on p-ef-ch-su00-gpu17-h1-aq-2f-2-cm-281-9-8u-0-17 device 5 [0000:ab:00] NVIDIA H100 80GB HBM3 # Rank 126 Group 0 Pid 3218589 on p-ef-ch-su00-gpu17-h1-aq-2f-2-cm-281-9-8u-0-17 device 6 [0000:ba:00] NVIDIA H100 80GB HBM3 # Rank 127 Group 0 Pid 3218590 on p-ef-ch-su00-gpu17-h1-aq-2f-2-cm-281-9-8u-0-17 device 7 [0000:db:00] NVIDIA H100 80GB HBM3 NCCL version 2.20.5+cuda12.4 # # out-of-place in-place # size count type redop root time algbw busbw #wrong time algbw busbw #wrong # (B) (elements) (us) (GB/s) (GB/s) (us) (GB/s) (GB/s) 17179869184 4294967296 float sum -1 87708 195.88 388.69 0 87618 196.08 389.09 0 # Out of bounds values : 0 OK # Avg bus bandwidth : 388.89 # 参与的服务器数量=16,总卡数=128 #