Heuristic summary (2025-09-27_103734): 1) GPUs at 100% util with low mem? GPU util=+---------------------------------------------------------------------------------------+ mem_usage? check raw files GPU util=| NVIDIA-SMI 535.161.08 Driver Version: 535.161.08 CUDA Version: 12.2 | mem_usage? check raw files GPU util=|-----------------------------------------+----------------------+----------------------+ mem_usage? check raw files GPU util=| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | mem_usage? check raw files GPU util=| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | mem_usage? check raw files GPU util=| | | MIG M. | mem_usage? check raw files GPU util=|=========================================+======================+======================| mem_usage? check raw files GPU util=| 0 NVIDIA H100 80GB HBM3 On | 00000000:19:00.0 Off | 0 | mem_usage? check raw files GPU util=| N/A 58C P0 697W / 700W | 5619MiB / 81559MiB | 100% Default | mem_usage? check raw files GPU util=| | | Disabled | mem_usage? check raw files GPU util=+-----------------------------------------+----------------------+----------------------+ mem_usage? check raw files GPU util=| 1 NVIDIA H100 80GB HBM3 On | 00000000:3B:00.0 Off | 0 | mem_usage? check raw files GPU util=| N/A 52C P0 698W / 700W | 5619MiB / 81559MiB | 100% Default | mem_usage? check raw files GPU util=| | | Disabled | mem_usage? check raw files GPU util=+-----------------------------------------+----------------------+----------------------+ mem_usage? check raw files GPU util=| 2 NVIDIA H100 80GB HBM3 On | 00000000:4C:00.0 Off | 0 | mem_usage? check raw files GPU util=| N/A 52C P0 698W / 700W | 5619MiB / 81559MiB | 100% Default | mem_usage? check raw files GPU util=| | | Disabled | mem_usage? check raw files GPU util=+-----------------------------------------+----------------------+----------------------+ mem_usage? check raw files GPU util=| 3 NVIDIA H100 80GB HBM3 On | 00000000:5D:00.0 Off | 0 | mem_usage? check raw files GPU util=| N/A 59C P0 698W / 700W | 5619MiB / 81559MiB | 100% Default | mem_usage? check raw files GPU util=| | | Disabled | mem_usage? check raw files GPU util=+-----------------------------------------+----------------------+----------------------+ mem_usage? check raw files GPU util=| 4 NVIDIA H100 80GB HBM3 On | 00000000:9B:00.0 Off | 0 | mem_usage? check raw files GPU util=| N/A 60C P0 698W / 700W | 5619MiB / 81559MiB | 100% Default | mem_usage? check raw files GPU util=| | | Disabled | mem_usage? check raw files GPU util=+-----------------------------------------+----------------------+----------------------+ mem_usage? check raw files GPU util=| 5 NVIDIA H100 80GB HBM3 On | 00000000:BB:00.0 Off | 0 | mem_usage? check raw files GPU util=| N/A 53C P0 698W / 700W | 5619MiB / 81559MiB | 100% Default | mem_usage? check raw files GPU util=| | | Disabled | mem_usage? check raw files GPU util=+-----------------------------------------+----------------------+----------------------+ mem_usage? check raw files GPU util=| 6 NVIDIA H100 80GB HBM3 On | 00000000:CB:00.0 Off | 0 | mem_usage? check raw files GPU util=| N/A 59C P0 697W / 700W | 5619MiB / 81559MiB | 100% Default | mem_usage? check raw files GPU util=| | | Disabled | mem_usage? check raw files GPU util=+-----------------------------------------+----------------------+----------------------+ mem_usage? check raw files GPU util=| 7 NVIDIA H100 80GB HBM3 On | 00000000:DB:00.0 Off | 0 | mem_usage? check raw files GPU util=| N/A 51C P0 698W / 700W | 5619MiB / 81559MiB | 100% Default | mem_usage? check raw files GPU util=| | | Disabled | mem_usage? check raw files GPU util=+-----------------------------------------+----------------------+----------------------+ mem_usage? check raw files GPU util= mem_usage? check raw files GPU util=+---------------------------------------------------------------------------------------+ mem_usage? check raw files GPU util=| Processes: | mem_usage? check raw files GPU util=| GPU GI CI PID Type Process name GPU Memory | mem_usage? check raw files GPU util=| ID ID Usage | mem_usage? check raw files GPU util=|=======================================================================================| mem_usage? check raw files GPU util=| 0 N/A N/A 3599569 C /etc/system 5610MiB | mem_usage? check raw files GPU util=| 1 N/A N/A 3599569 C /etc/system 5610MiB | mem_usage? check raw files GPU util=| 2 N/A N/A 3599569 C /etc/system 5610MiB | mem_usage? check raw files GPU util=| 3 N/A N/A 3599569 C /etc/system 5610MiB | mem_usage? check raw files GPU util=| 4 N/A N/A 3599569 C /etc/system 5610MiB | mem_usage? check raw files GPU util=| 5 N/A N/A 3599569 C /etc/system 5610MiB | mem_usage? check raw files GPU util=| 6 N/A N/A 3599569 C /etc/system 5610MiB | mem_usage? check raw files GPU util=| 7 N/A N/A 3599569 C /etc/system 5610MiB | mem_usage? check raw files GPU util=+---------------------------------------------------------------------------------------+ mem_usage? check raw files 2) Any compute apps reported? gpu_uuid, pid, process_name, used_gpu_memory [MiB] GPU-c58ae247-e8ab-7e82-2a70-985217ce0668, 3599569, /etc/system, 5610 MiB GPU-21ac6c7d-7c7d-b092-82f8-2cbfa90bc9ea, 3599569, /etc/system, 5610 MiB GPU-c9702494-4e51-248c-e543-b590f811a9b0, 3599569, /etc/system, 5610 MiB GPU-90f8676b-993f-15cb-a068-8d1046c97acc, 3599569, /etc/system, 5610 MiB GPU-cce068a9-09b9-e975-51d8-3a1c08598b93, 3599569, /etc/system, 5610 MiB GPU-be46e941-f5eb-48a9-9e87-891afa1cb340, 3599569, /etc/system, 5610 MiB GPU-744ae3a0-5f1f-2c83-0e7f-434b9402e304, 3599569, /etc/system, 5610 MiB GPU-884506b7-1154-b58e-7249-b10995e8f1aa, 3599569, /etc/system, 5610 MiB 3) Any device users via lsof/fuser? nv-hosten 3903 root 3u CHR 195,255 0t0 1760 /dev/nvidiactl nv-hosten 3903 root 4u CHR 195,0 0t0 1762 /dev/nvidia0 nv-hosten 3903 root 5u CHR 195,1 0t0 1767 /dev/nvidia1 nv-hosten 3903 root 6u CHR 195,2 0t0 1774 /dev/nvidia2 nv-hosten 3903 root 7u CHR 195,3 0t0 1782 /dev/nvidia3 nv-hosten 3903 root 8u CHR 195,4 0t0 1789 /dev/nvidia4 nv-hosten 3903 root 9u CHR 195,5 0t0 1815 /dev/nvidia5 nv-hosten 3903 root 10u CHR 195,6 0t0 1829 /dev/nvidia6 nv-hosten 3903 root 11u CHR 195,7 0t0 1842 /dev/nvidia7 nv-hosten 3903 root 12u CHR 502,0 0t0 1920 /dev/nvidia-uvm nv-hosten 3903 root 15u CHR 195,0 0t0 1762 /dev/nvidia0 nv-hosten 3903 root 16u CHR 195,0 0t0 1762 /dev/nvidia0 nv-hosten 3903 root 17u CHR 195,1 0t0 1767 /dev/nvidia1 nv-hosten 3903 root 18u CHR 195,1 0t0 1767 /dev/nvidia1 nv-hosten 3903 root 19u CHR 195,2 0t0 1774 /dev/nvidia2 nv-hosten 3903 root 20u CHR 195,2 0t0 1774 /dev/nvidia2 nv-hosten 3903 root 21u CHR 195,3 0t0 1782 /dev/nvidia3 nv-hosten 3903 root 22u CHR 195,3 0t0 1782 /dev/nvidia3 nv-hosten 3903 root 23u CHR 195,4 0t0 1789 /dev/nvidia4 nv-hosten 3903 root 24u CHR 195,4 0t0 1789 /dev/nvidia4 nv-hosten 3903 root 25u CHR 195,5 0t0 1815 /dev/nvidia5 nv-hosten 3903 root 26u CHR 195,5 0t0 1815 /dev/nvidia5 nv-hosten 3903 root 27u CHR 195,6 0t0 1829 /dev/nvidia6 nv-hosten 3903 root 28u CHR 195,6 0t0 1829 /dev/nvidia6 nv-hosten 3903 root 29u CHR 195,7 0t0 1842 /dev/nvidia7 nv-hosten 3903 root 30u CHR 195,7 0t0 1842 /dev/nvidia7 nv-hosten 3903 root 31u CHR 195,0 0t0 1762 /dev/nvidia0 nv-hosten 3903 root 32u CHR 195,1 0t0 1767 /dev/nvidia1 nv-hosten 3903 root 33u CHR 195,2 0t0 1774 /dev/nvidia2 nv-hosten 3903 root 34u CHR 195,3 0t0 1782 /dev/nvidia3 nv-hosten 3903 root 35u CHR 195,4 0t0 1789 /dev/nvidia4 nv-hosten 3903 root 36u CHR 195,5 0t0 1815 /dev/nvidia5 nv-hosten 3903 root 37u CHR 195,6 0t0 1829 /dev/nvidia6 nv-hosten 3903 root 38u CHR 195,7 0t0 1842 /dev/nvidia7 nv-fabric 4509 root 4u CHR 505,0 0t0 1795 /dev/nvidia-nvswitch0 nv-fabric 4509 root 5u CHR 505,1 0t0 1797 /dev/nvidia-nvswitch1 nv-fabric 4509 root 7u CHR 505,2 0t0 1799 /dev/nvidia-nvswitch2 nv-fabric 4509 root 9u CHR 505,3 0t0 1801 /dev/nvidia-nvswitch3 nv-fabric 4509 root 16u CHR 506,0 0t0 1809 /dev/nvidia-nvlink nv-fabric 4509 root 29u CHR 505,0 0t0 1795 /dev/nvidia-nvswitch0 nv-fabric 4509 root 30u CHR 505,0 0t0 1795 /dev/nvidia-nvswitch0 nv-fabric 4509 root 31u CHR 505,0 0t0 1795 /dev/nvidia-nvswitch0 nv-fabric 4509 root 32u CHR 505,1 0t0 1797 /dev/nvidia-nvswitch1 nv-fabric 4509 root 33u CHR 505,1 0t0 1797 /dev/nvidia-nvswitch1 nv-fabric 4509 root 34u CHR 505,1 0t0 1797 /dev/nvidia-nvswitch1 nv-fabric 4509 root 35u CHR 505,2 0t0 1799 /dev/nvidia-nvswitch2 nv-fabric 4509 root 36u CHR 505,2 0t0 1799 /dev/nvidia-nvswitch2 nv-fabric 4509 root 37u CHR 505,2 0t0 1799 /dev/nvidia-nvswitch2 nv-fabric 4509 root 38u CHR 505,3 0t0 1801 /dev/nvidia-nvswitch3 nv-fabric 4509 root 39u CHR 505,3 0t0 1801 /dev/nvidia-nvswitch3 nv-fabric 4509 root 40u CHR 505,3 0t0 1801 /dev/nvidia-nvswitch3 dcgm-expo 8645 root 3u CHR 195,255 0t0 1760 /dev/nvidiactl dcgm-expo 8645 root 7u CHR 195,0 0t0 1762 /dev/nvidia0 dcgm-expo 8645 root 8u CHR 195,1 0t0 1767 /dev/nvidia1 dcgm-expo 8645 root 9u CHR 195,2 0t0 1774 /dev/nvidia2 dcgm-expo 8645 root 10u CHR 195,3 0t0 1782 /dev/nvidia3 dcgm-expo 8645 root 11u CHR 195,4 0t0 1789 /dev/nvidia4 dcgm-expo 8645 root 12u CHR 195,5 0t0 1815 /dev/nvidia5 dcgm-expo 8645 root 13u CHR 195,6 0t0 1829 /dev/nvidia6 dcgm-expo 8645 root 14u CHR 195,7 0t0 1842 /dev/nvidia7 dcgm-expo 8645 root 15u CHR 502,0 0t0 1920 /dev/nvidia-uvm dcgm-expo 8645 root 16u CHR 195,0 0t0 1762 /dev/nvidia0 dcgm-expo 8645 root 17u CHR 195,0 0t0 1762 /dev/nvidia0 dcgm-expo 8645 root 18u CHR 195,1 0t0 1767 /dev/nvidia1 dcgm-expo 8645 root 19u CHR 195,1 0t0 1767 /dev/nvidia1 dcgm-expo 8645 root 20u CHR 195,2 0t0 1774 /dev/nvidia2 dcgm-expo 8645 root 21u CHR 195,2 0t0 1774 /dev/nvidia2 dcgm-expo 8645 root 22u CHR 195,3 0t0 1782 /dev/nvidia3 dcgm-expo 8645 root 23u CHR 195,3 0t0 1782 /dev/nvidia3 dcgm-expo 8645 root 24u CHR 195,4 0t0 1789 /dev/nvidia4 dcgm-expo 8645 root 25u CHR 195,4 0t0 1789 /dev/nvidia4 dcgm-expo 8645 root 26u CHR 195,5 0t0 1815 /dev/nvidia5 dcgm-expo 8645 root 27u CHR 195,5 0t0 1815 /dev/nvidia5 dcgm-expo 8645 root 28u CHR 195,6 0t0 1829 /dev/nvidia6 dcgm-expo 8645 root 29u CHR 195,6 0t0 1829 /dev/nvidia6 dcgm-expo 8645 root 30u CHR 195,7 0t0 1842 /dev/nvidia7 dcgm-expo 8645 root 31u CHR 195,7 0t0 1842 /dev/nvidia7 dcgm-expo 8645 root 32u CHR 195,0 0t0 1762 /dev/nvidia0 dcgm-expo 8645 root 33u CHR 195,1 0t0 1767 /dev/nvidia1 dcgm-expo 8645 root 34u CHR 195,2 0t0 1774 /dev/nvidia2 dcgm-expo 8645 root 35u CHR 195,3 0t0 1782 /dev/nvidia3 dcgm-expo 8645 root 36u CHR 195,4 0t0 1789 /dev/nvidia4 dcgm-expo 8645 root 37u CHR 195,5 0t0 1815 /dev/nvidia5 dcgm-expo 8645 root 38u CHR 195,6 0t0 1829 /dev/nvidia6 dcgm-expo 8645 root 39u CHR 195,7 0t0 1842 /dev/nvidia7 nvidia-de 11377 root 10u CHR 195,255 0t0 1760 /dev/nvidiactl nvidia-de 11377 root 11u CHR 195,0 0t0 1762 /dev/nvidia0 nvidia-de 11377 root 12u CHR 195,1 0t0 1767 /dev/nvidia1 nvidia-de 11377 root 13u CHR 195,2 0t0 1774 /dev/nvidia2 nvidia-de 11377 root 14u CHR 195,3 0t0 1782 /dev/nvidia3 nvidia-de 11377 root 15u CHR 195,4 0t0 1789 /dev/nvidia4 nvidia-de 11377 root 16u CHR 195,5 0t0 1815 /dev/nvidia5 nvidia-de 11377 root 17u CHR 195,6 0t0 1829 /dev/nvidia6 nvidia-de 11377 root 18u CHR 195,7 0t0 1842 /dev/nvidia7 nvidia-de 11377 root 22u CHR 195,0 0t0 1762 /dev/nvidia0 nvidia-de 11377 root 23u CHR 195,0 0t0 1762 /dev/nvidia0 nvidia-de 11377 root 24u CHR 195,1 0t0 1767 /dev/nvidia1 nvidia-de 11377 root 25u CHR 195,1 0t0 1767 /dev/nvidia1 nvidia-de 11377 root 26u CHR 195,2 0t0 1774 /dev/nvidia2 nvidia-de 11377 root 27u CHR 195,2 0t0 1774 /dev/nvidia2 nvidia-de 11377 root 28u CHR 195,3 0t0 1782 /dev/nvidia3 nvidia-de 11377 root 29u CHR 195,3 0t0 1782 /dev/nvidia3 nvidia-de 11377 root 30u CHR 195,4 0t0 1789 /dev/nvidia4 nvidia-de 11377 root 31u CHR 195,4 0t0 1789 /dev/nvidia4 nvidia-de 11377 root 32u CHR 195,5 0t0 1815 /dev/nvidia5 nvidia-de 11377 root 33u CHR 195,5 0t0 1815 /dev/nvidia5 nvidia-de 11377 root 34u CHR 195,6 0t0 1829 /dev/nvidia6 nvidia-de 11377 root 35u CHR 195,6 0t0 1829 /dev/nvidia6 nvidia-de 11377 root 36u CHR 195,7 0t0 1842 /dev/nvidia7 nvidia-de 11377 root 37u CHR 195,7 0t0 1842 /dev/nvidia7 nvidia-de 11377 root 40u CHR 195,6 0t0 1829 /dev/nvidia6 nvidia-de 11377 root 41u CHR 195,7 0t0 1842 /dev/nvidia7 nvidia-de 11377 root 42u CHR 195,0 0t0 1762 /dev/nvidia0 nvidia-de 11377 root 43u CHR 195,1 0t0 1767 /dev/nvidia1 nvidia-de 11377 root 44u CHR 195,2 0t0 1774 /dev/nvidia2 nvidia-de 11377 root 45u CHR 195,3 0t0 1782 /dev/nvidia3 nvidia-de 11377 root 46u CHR 195,4 0t0 1789 /dev/nvidia4 nvidia-de 11377 root 47u CHR 195,5 0t0 1815 /dev/nvidia5 system 3599569 root mem CHR 195,0 1762 /dev/nvidia0 system 3599569 root mem CHR 195,255 1760 /dev/nvidiactl system 3599569 root mem CHR 502,0 1920 /dev/nvidia-uvm system 3599569 root mem CHR 195,1 1767 /dev/nvidia1 system 3599569 root mem CHR 195,2 1774 /dev/nvidia2 system 3599569 root mem CHR 195,3 1782 /dev/nvidia3 system 3599569 root mem CHR 195,4 1789 /dev/nvidia4 system 3599569 root mem CHR 195,5 1815 /dev/nvidia5 system 3599569 root mem CHR 195,6 1829 /dev/nvidia6 system 3599569 root mem CHR 195,7 1842 /dev/nvidia7 system 3599569 root 3u CHR 195,255 0t0 1760 /dev/nvidiactl system 3599569 root 4u CHR 502,0 0t0 1920 /dev/nvidia-uvm system 3599569 root 5u CHR 195,0 0t0 1762 /dev/nvidia0 system 3599569 root 6u CHR 195,1 0t0 1767 /dev/nvidia1 system 3599569 root 7u CHR 195,2 0t0 1774 /dev/nvidia2 system 3599569 root 8u CHR 195,3 0t0 1782 /dev/nvidia3 system 3599569 root 9u CHR 195,4 0t0 1789 /dev/nvidia4 system 3599569 root 10u CHR 195,5 0t0 1815 /dev/nvidia5 system 3599569 root 11u CHR 195,6 0t0 1829 /dev/nvidia6 system 3599569 root 12u CHR 195,7 0t0 1842 /dev/nvidia7 system 3599569 root 13u CHR 195,0 0t0 1762 /dev/nvidia0 system 3599569 root 14u CHR 195,0 0t0 1762 /dev/nvidia0 system 3599569 root 15u CHR 195,1 0t0 1767 /dev/nvidia1 system 3599569 root 16u CHR 195,1 0t0 1767 /dev/nvidia1 system 3599569 root 17u CHR 195,2 0t0 1774 /dev/nvidia2 system 3599569 root 18u CHR 195,2 0t0 1774 /dev/nvidia2 system 3599569 root 19u CHR 195,3 0t0 1782 /dev/nvidia3 system 3599569 root 20u CHR 195,3 0t0 1782 /dev/nvidia3 system 3599569 root 21u CHR 195,4 0t0 1789 /dev/nvidia4 system 3599569 root 22u CHR 195,4 0t0 1789 /dev/nvidia4 system 3599569 root 23u CHR 195,5 0t0 1815 /dev/nvidia5 system 3599569 root 24u CHR 195,5 0t0 1815 /dev/nvidia5 system 3599569 root 25u CHR 195,6 0t0 1829 /dev/nvidia6 system 3599569 root 26u CHR 195,6 0t0 1829 /dev/nvidia6 system 3599569 root 27u CHR 195,7 0t0 1842 /dev/nvidia7 system 3599569 root 28u CHR 195,7 0t0 1842 /dev/nvidia7 system 3599569 root 32u CHR 195,0 0t0 1762 /dev/nvidia0 system 3599569 root 33u CHR 195,0 0t0 1762 /dev/nvidia0 system 3599569 root 34u CHR 195,0 0t0 1762 /dev/nvidia0 system 3599569 root 35u CHR 195,0 0t0 1762 /dev/nvidia0 system 3599569 root 36u CHR 195,0 0t0 1762 /dev/nvidia0 system 3599569 root 37u CHR 195,0 0t0 1762 /dev/nvidia0 system 3599569 root 38u CHR 195,0 0t0 1762 /dev/nvidia0 system 3599569 root 39u CHR 195,0 0t0 1762 /dev/nvidia0 system 3599569 root 41u CHR 195,0 0t0 1762 /dev/nvidia0 system 3599569 root 42u CHR 195,0 0t0 1762 /dev/nvidia0 system 3599569 root 43u CHR 195,0 0t0 1762 /dev/nvidia0 system 3599569 root 44u CHR 195,0 0t0 1762 /dev/nvidia0 system 3599569 root 45u CHR 195,0 0t0 1762 /dev/nvidia0 system 3599569 root 46u CHR 195,0 0t0 1762 /dev/nvidia0 system 3599569 root 47u CHR 195,0 0t0 1762 /dev/nvidia0 system 3599569 root 48u CHR 195,0 0t0 1762 /dev/nvidia0 system 3599569 root 52u CHR 195,1 0t0 1767 /dev/nvidia1 system 3599569 root 53u CHR 195,1 0t0 1767 /dev/nvidia1 system 3599569 root 54u CHR 195,1 0t0 1767 /dev/nvidia1 system 3599569 root 55u CHR 195,1 0t0 1767 /dev/nvidia1 system 3599569 root 56u CHR 195,1 0t0 1767 /dev/nvidia1 system 3599569 root 57u CHR 195,1 0t0 1767 /dev/nvidia1 system 3599569 root 58u CHR 195,1 0t0 1767 /dev/nvidia1 system 3599569 root 59u CHR 195,1 0t0 1767 /dev/nvidia1 system 3599569 root 61u CHR 195,1 0t0 1767 /dev/nvidia1 system 3599569 root 62u CHR 195,1 0t0 1767 /dev/nvidia1 system 3599569 root 63u CHR 195,1 0t0 1767 /dev/nvidia1 system 3599569 root 64u CHR 195,1 0t0 1767 /dev/nvidia1 system 3599569 root 65u CHR 195,1 0t0 1767 /dev/nvidia1 system 3599569 root 66u CHR 195,1 0t0 1767 /dev/nvidia1 system 3599569 root 67u CHR 195,1 0t0 1767 /dev/nvidia1 system 3599569 root 68u CHR 195,1 0t0 1767 /dev/nvidia1 system 3599569 root 70u CHR 195,2 0t0 1774 /dev/nvidia2 system 3599569 root 71u CHR 195,2 0t0 1774 /dev/nvidia2 system 3599569 root 72u CHR 195,2 0t0 1774 /dev/nvidia2 system 3599569 root 73u CHR 195,2 0t0 1774 /dev/nvidia2 system 3599569 root 74u CHR 195,2 0t0 1774 /dev/nvidia2 system 3599569 root 75u CHR 195,2 0t0 1774 /dev/nvidia2 system 3599569 root 76u CHR 195,2 0t0 1774 /dev/nvidia2 system 3599569 root 77u CHR 195,2 0t0 1774 /dev/nvidia2 system 3599569 root 79u CHR 195,2 0t0 1774 /dev/nvidia2 system 3599569 root 80u CHR 195,2 0t0 1774 /dev/nvidia2 system 3599569 root 81u CHR 195,2 0t0 1774 /dev/nvidia2 system 3599569 root 82u CHR 195,2 0t0 1774 /dev/nvidia2 system 3599569 root 83u CHR 195,2 0t0 1774 /dev/nvidia2 system 3599569 root 84u CHR 195,2 0t0 1774 /dev/nvidia2 system 3599569 root 85u CHR 195,2 0t0 1774 /dev/nvidia2 system 3599569 root 86u CHR 195,2 0t0 1774 /dev/nvidia2 system 3599569 root 88u CHR 195,3 0t0 1782 /dev/nvidia3 system 3599569 root 89u CHR 195,3 0t0 1782 /dev/nvidia3 system 3599569 root 90u CHR 195,3 0t0 1782 /dev/nvidia3 system 3599569 root 91u CHR 195,3 0t0 1782 /dev/nvidia3 system 3599569 root 92u CHR 195,3 0t0 1782 /dev/nvidia3 system 3599569 root 93u CHR 195,3 0t0 1782 /dev/nvidia3 system 3599569 root 94u CHR 195,3 0t0 1782 /dev/nvidia3 system 3599569 root 95u CHR 195,3 0t0 1782 /dev/nvidia3 system 3599569 root 97u CHR 195,3 0t0 1782 /dev/nvidia3 system 3599569 root 98u CHR 195,3 0t0 1782 /dev/nvidia3 system 3599569 root 99u CHR 195,3 0t0 1782 /dev/nvidia3 system 3599569 root 100u CHR 195,3 0t0 1782 /dev/nvidia3 system 3599569 root 101u CHR 195,3 0t0 1782 /dev/nvidia3 system 3599569 root 102u CHR 195,3 0t0 1782 /dev/nvidia3 system 3599569 root 103u CHR 195,3 0t0 1782 /dev/nvidia3 system 3599569 root 104u CHR 195,3 0t0 1782 /dev/nvidia3 system 3599569 root 106u CHR 195,4 0t0 1789 /dev/nvidia4 system 3599569 root 107u CHR 195,4 0t0 1789 /dev/nvidia4 system 3599569 root 108u CHR 195,4 0t0 1789 /dev/nvidia4 system 3599569 root 109u CHR 195,4 0t0 1789 /dev/nvidia4 system 3599569 root 110u CHR 195,4 0t0 1789 /dev/nvidia4 system 3599569 root 111u CHR 195,4 0t0 1789 /dev/nvidia4 system 3599569 root 112u CHR 195,4 0t0 1789 /dev/nvidia4 system 3599569 root 113u CHR 195,4 0t0 1789 /dev/nvidia4 system 3599569 root 115u CHR 195,4 0t0 1789 /dev/nvidia4 system 3599569 root 116u CHR 195,4 0t0 1789 /dev/nvidia4 system 3599569 root 117u CHR 195,4 0t0 1789 /dev/nvidia4 system 3599569 root 118u CHR 195,4 0t0 1789 /dev/nvidia4 system 3599569 root 119u CHR 195,4 0t0 1789 /dev/nvidia4 system 3599569 root 120u CHR 195,4 0t0 1789 /dev/nvidia4 system 3599569 root 121u CHR 195,4 0t0 1789 /dev/nvidia4 system 3599569 root 122u CHR 195,4 0t0 1789 /dev/nvidia4 system 3599569 root 124u CHR 195,5 0t0 1815 /dev/nvidia5 system 3599569 root 125u CHR 195,5 0t0 1815 /dev/nvidia5 system 3599569 root 126u CHR 195,5 0t0 1815 /dev/nvidia5 system 3599569 root 127u CHR 195,5 0t0 1815 /dev/nvidia5 system 3599569 root 128u CHR 195,5 0t0 1815 /dev/nvidia5 system 3599569 root 129u CHR 195,5 0t0 1815 /dev/nvidia5 system 3599569 root 130u CHR 195,5 0t0 1815 /dev/nvidia5 system 3599569 root 131u CHR 195,5 0t0 1815 /dev/nvidia5 system 3599569 root 133u CHR 195,5 0t0 1815 /dev/nvidia5 system 3599569 root 134u CHR 195,5 0t0 1815 /dev/nvidia5 system 3599569 root 135u CHR 195,5 0t0 1815 /dev/nvidia5 system 3599569 root 136u CHR 195,5 0t0 1815 /dev/nvidia5 system 3599569 root 137u CHR 195,5 0t0 1815 /dev/nvidia5 system 3599569 root 138u CHR 195,5 0t0 1815 /dev/nvidia5 system 3599569 root 139u CHR 195,5 0t0 1815 /dev/nvidia5 system 3599569 root 140u CHR 195,5 0t0 1815 /dev/nvidia5 system 3599569 root 142u CHR 195,6 0t0 1829 /dev/nvidia6 system 3599569 root 143u CHR 195,6 0t0 1829 /dev/nvidia6 system 3599569 root 144u CHR 195,6 0t0 1829 /dev/nvidia6 system 3599569 root 145u CHR 195,6 0t0 1829 /dev/nvidia6 system 3599569 root 146u CHR 195,6 0t0 1829 /dev/nvidia6 system 3599569 root 147u CHR 195,6 0t0 1829 /dev/nvidia6 system 3599569 root 148u CHR 195,6 0t0 1829 /dev/nvidia6 system 3599569 root 149u CHR 195,6 0t0 1829 /dev/nvidia6 system 3599569 root 151u CHR 195,6 0t0 1829 /dev/nvidia6 system 3599569 root 152u CHR 195,6 0t0 1829 /dev/nvidia6 system 3599569 root 153u CHR 195,6 0t0 1829 /dev/nvidia6 system 3599569 root 154u CHR 195,6 0t0 1829 /dev/nvidia6 system 3599569 root 155u CHR 195,6 0t0 1829 /dev/nvidia6 system 3599569 root 156u CHR 195,6 0t0 1829 /dev/nvidia6 system 3599569 root 157u CHR 195,6 0t0 1829 /dev/nvidia6 system 3599569 root 158u CHR 195,6 0t0 1829 /dev/nvidia6 system 3599569 root 160u CHR 195,7 0t0 1842 /dev/nvidia7 system 3599569 root 161u CHR 195,7 0t0 1842 /dev/nvidia7 system 3599569 root 162u CHR 195,7 0t0 1842 /dev/nvidia7 system 3599569 root 163u CHR 195,7 0t0 1842 /dev/nvidia7 system 3599569 root 164u CHR 195,7 0t0 1842 /dev/nvidia7 system 3599569 root 165u CHR 195,7 0t0 1842 /dev/nvidia7 system 3599569 root 166u CHR 195,7 0t0 1842 /dev/nvidia7 system 3599569 root 167u CHR 195,7 0t0 1842 /dev/nvidia7 system 3599569 root 169u CHR 195,7 0t0 1842 /dev/nvidia7 system 3599569 root 170u CHR 195,7 0t0 1842 /dev/nvidia7 system 3599569 root 171u CHR 195,7 0t0 1842 /dev/nvidia7 system 3599569 root 172u CHR 195,7 0t0 1842 /dev/nvidia7 system 3599569 root 173u CHR 195,7 0t0 1842 /dev/nvidia7 system 3599569 root 174u CHR 195,7 0t0 1842 /dev/nvidia7 system 3599569 root 175u CHR 195,7 0t0 1842 /dev/nvidia7 system 3599569 root 176u CHR 195,7 0t0 1842 /dev/nvidia7 system 3599569 root 181u CHR 195,255 0t0 1760 /dev/nvidiactl system 3599569 root 183u CHR 195,0 0t0 1762 /dev/nvidia0 system 3599569 root 184u CHR 195,1 0t0 1767 /dev/nvidia1 system 3599569 root 185u CHR 195,2 0t0 1774 /dev/nvidia2 system 3599569 root 186u CHR 195,3 0t0 1782 /dev/nvidia3 system 3599569 root 187u CHR 195,4 0t0 1789 /dev/nvidia4 system 3599569 root 188u CHR 195,5 0t0 1815 /dev/nvidia5 system 3599569 root 189u CHR 195,6 0t0 1829 /dev/nvidia6 system 3599569 root 190u CHR 195,7 0t0 1842 /dev/nvidia7 system 3599569 root 191u CHR 502,0 0t0 1920 /dev/nvidia-uvm system 3599569 root 194u CHR 195,0 0t0 1762 /dev/nvidia0 system 3599569 root 195u CHR 195,0 0t0 1762 /dev/nvidia0 system 3599569 root 196u CHR 195,1 0t0 1767 /dev/nvidia1 system 3599569 root 197u CHR 195,1 0t0 1767 /dev/nvidia1 system 3599569 root 198u CHR 195,2 0t0 1774 /dev/nvidia2 system 3599569 root 199u CHR 195,2 0t0 1774 /dev/nvidia2 system 3599569 root 200u CHR 195,3 0t0 1782 /dev/nvidia3 system 3599569 root 201u CHR 195,3 0t0 1782 /dev/nvidia3 system 3599569 root 202u CHR 195,4 0t0 1789 /dev/nvidia4 system 3599569 root 203u CHR 195,4 0t0 1789 /dev/nvidia4 system 3599569 root 204u CHR 195,5 0t0 1815 /dev/nvidia5 system 3599569 root 205u CHR 195,5 0t0 1815 /dev/nvidia5 system 3599569 root 206u CHR 195,6 0t0 1829 /dev/nvidia6 system 3599569 root 207u CHR 195,6 0t0 1829 /dev/nvidia6 system 3599569 root 208u CHR 195,7 0t0 1842 /dev/nvidia7 system 3599569 root 209u CHR 195,7 0t0 1842 /dev/nvidia7 system 3599569 root 212u CHR 195,255 0t0 1760 /dev/nvidiactl system 3599569 root 213u CHR 502,0 0t0 1920 /dev/nvidia-uvm system 3599569 root 214u CHR 195,0 0t0 1762 /dev/nvidia0 system 3599569 root 215u CHR 195,1 0t0 1767 /dev/nvidia1 system 3599569 root 216u CHR 195,2 0t0 1774 /dev/nvidia2 system 3599569 root 217u CHR 195,3 0t0 1782 /dev/nvidia3 system 3599569 root 218u CHR 195,4 0t0 1789 /dev/nvidia4 system 3599569 root 219u CHR 195,5 0t0 1815 /dev/nvidia5 system 3599569 root 220u CHR 195,6 0t0 1829 /dev/nvidia6 system 3599569 root 221u CHR 195,7 0t0 1842 /dev/nvidia7 system 3599569 root 222u CHR 195,0 0t0 1762 /dev/nvidia0 system 3599569 root 223u CHR 195,0 0t0 1762 /dev/nvidia0 system 3599569 root 224u CHR 195,1 0t0 1767 /dev/nvidia1 system 3599569 root 225u CHR 195,1 0t0 1767 /dev/nvidia1 system 3599569 root 226u CHR 195,2 0t0 1774 /dev/nvidia2 system 3599569 root 227u CHR 195,2 0t0 1774 /dev/nvidia2 system 3599569 root 228u CHR 195,3 0t0 1782 /dev/nvidia3 system 3599569 root 229u CHR 195,3 0t0 1782 /dev/nvidia3 system 3599569 root 230u CHR 195,4 0t0 1789 /dev/nvidia4 system 3599569 root 231u CHR 195,4 0t0 1789 /dev/nvidia4 system 3599569 root 232u CHR 195,5 0t0 1815 /dev/nvidia5 system 3599569 root 233u CHR 195,5 0t0 1815 /dev/nvidia5 system 3599569 root 234u CHR 195,6 0t0 1829 /dev/nvidia6 system 3599569 root 235u CHR 195,6 0t0 1829 /dev/nvidia6 system 3599569 root 236u CHR 195,7 0t0 1842 /dev/nvidia7 system 3599569 root 237u CHR 195,7 0t0 1842 /dev/nvidia7 4) Any suspicious processes? root 3621 0.0 0.0 0 0 ? S Jun20 0:00 [nvidia-modeset/] root 3622 0.0 0.0 0 0 ? S Jun20 0:00 [nvidia-modeset/] root 3903 0.0 0.0 403200 21276 ? Ssl Jun20 9:45 /usr/bin/nv-hostengine -n --service-account nvidia-dcgm root 3977 0.0 0.0 0 0 ? S Jun20 0:37 [irq/1361-nvidia] root 3978 0.0 0.0 0 0 ? S Jun20 0:00 [irq/1362-nvidia] root 3979 0.0 0.0 0 0 ? S Jun20 0:00 [irq/1363-nvidia] root 3980 0.0 0.0 0 0 ? S Jun20 0:00 [irq/1364-nvidia] root 3981 0.0 0.0 0 0 ? S Jun20 0:00 [irq/1365-nvidia] root 3982 0.0 0.0 0 0 ? S Jun20 0:00 [irq/1366-nvidia] root 3983 0.0 0.0 0 0 ? S Jun20 0:00 [irq/1367-nvidia] root 3984 0.0 0.0 0 0 ? S Jun20 0:00 [irq/1368-nvidia] root 3985 0.0 0.0 0 0 ? S Jun20 0:00 [nvidia] root 4247 0.0 0.0 0 0 ? S Jun20 0:36 [irq/1379-nvidia] root 4248 0.0 0.0 0 0 ? S Jun20 0:00 [irq/1380-nvidia] root 4249 0.0 0.0 0 0 ? S Jun20 0:00 [irq/1381-nvidia] root 4250 0.0 0.0 0 0 ? S Jun20 0:00 [irq/1382-nvidia] root 4251 0.0 0.0 0 0 ? S Jun20 0:00 [irq/1383-nvidia] root 4252 0.0 0.0 0 0 ? S Jun20 0:00 [irq/1384-nvidia] root 4253 0.0 0.0 0 0 ? S Jun20 0:00 [irq/1385-nvidia] root 4254 0.0 0.0 0 0 ? S Jun20 0:00 [irq/1386-nvidia] root 4255 0.0 0.0 0 0 ? S Jun20 0:00 [nvidia] root 4314 0.0 0.0 0 0 ? S Jun20 0:36 [irq/1387-nvidia] root 4315 0.0 0.0 0 0 ? S Jun20 0:00 [irq/1388-nvidia] root 4316 0.0 0.0 0 0 ? S Jun20 0:00 [irq/1389-nvidia] root 4317 0.0 0.0 0 0 ? S Jun20 0:00 [irq/1390-nvidia] root 4318 0.0 0.0 0 0 ? S Jun20 0:00 [irq/1391-nvidia] root 4319 0.0 0.0 0 0 ? S Jun20 0:00 [irq/1392-nvidia] root 4320 0.0 0.0 0 0 ? S Jun20 0:00 [irq/1393-nvidia] root 4321 0.0 0.0 0 0 ? S Jun20 0:00 [irq/1394-nvidia] root 4322 0.0 0.0 0 0 ? S Jun20 0:00 [nvidia] root 4465 0.0 0.0 0 0 ? S Jun20 0:36 [irq/1395-nvidia] root 4466 0.0 0.0 0 0 ? S Jun20 0:00 [irq/1396-nvidia] root 4467 0.0 0.0 0 0 ? S Jun20 0:00 [irq/1397-nvidia] root 4468 0.0 0.0 0 0 ? S Jun20 0:00 [irq/1398-nvidia] root 4469 0.0 0.0 0 0 ? S Jun20 0:00 [irq/1399-nvidia] root 4470 0.0 0.0 0 0 ? S Jun20 0:00 [irq/1400-nvidia] root 4471 0.0 0.0 0 0 ? S Jun20 0:00 [irq/1401-nvidia] root 4472 0.0 0.0 0 0 ? S Jun20 0:00 [irq/1402-nvidia] root 4473 0.0 0.0 0 0 ? S Jun20 0:00 [nvidia] root 4509 0.0 0.0 1240624 9840 ? Sl Jun20 37:17 /usr/bin/nv-fabricmanager -c /usr/share/nvidia/nvswitch/fabricmanager.cfg root 4665 0.0 0.0 0 0 ? S Jun20 0:37 [irq/1403-nvidia] root 4666 0.0 0.0 0 0 ? S Jun20 0:00 [irq/1404-nvidia] root 4667 0.0 0.0 0 0 ? S Jun20 0:00 [irq/1405-nvidia] root 4668 0.0 0.0 0 0 ? S Jun20 0:00 [irq/1406-nvidia] root 4669 0.0 0.0 0 0 ? S Jun20 0:00 [irq/1407-nvidia] root 4670 0.0 0.0 0 0 ? S Jun20 0:00 [irq/1408-nvidia] root 4671 0.0 0.0 0 0 ? S Jun20 0:00 [irq/1409-nvidia] root 4672 0.0 0.0 0 0 ? S Jun20 0:00 [irq/1410-nvidia] root 4673 0.0 0.0 0 0 ? S Jun20 0:00 [nvidia] root 4726 0.0 0.0 0 0 ? S Jun20 0:36 [irq/1411-nvidia] root 4727 0.0 0.0 0 0 ? S Jun20 0:00 [irq/1412-nvidia] root 4728 0.0 0.0 0 0 ? S Jun20 0:00 [irq/1413-nvidia] root 4729 0.0 0.0 0 0 ? S Jun20 0:00 [irq/1414-nvidia] root 4730 0.0 0.0 0 0 ? S Jun20 0:00 [irq/1415-nvidia] root 4731 0.0 0.0 0 0 ? S Jun20 0:00 [irq/1416-nvidia] root 4732 0.0 0.0 0 0 ? S Jun20 0:00 [irq/1417-nvidia] root 4733 0.0 0.0 0 0 ? S Jun20 0:00 [irq/1418-nvidia] root 4734 0.0 0.0 0 0 ? S Jun20 0:00 [nvidia] root 5095 0.0 0.0 0 0 ? S Jun20 0:36 [irq/1419-nvidia] root 5096 0.0 0.0 0 0 ? S Jun20 0:00 [irq/1420-nvidia] root 5097 0.0 0.0 0 0 ? S Jun20 0:00 [irq/1421-nvidia] root 5098 0.0 0.0 0 0 ? S Jun20 0:00 [irq/1422-nvidia] root 5099 0.0 0.0 0 0 ? S Jun20 0:00 [irq/1423-nvidia] root 5100 0.0 0.0 0 0 ? S Jun20 0:00 [irq/1424-nvidia] root 5101 0.0 0.0 0 0 ? S Jun20 0:00 [irq/1425-nvidia] root 5102 0.0 0.0 0 0 ? S Jun20 0:00 [irq/1426-nvidia] root 5103 0.0 0.0 0 0 ? S Jun20 0:00 [nvidia] root 5109 0.0 0.0 0 0 ? S Jun20 0:36 [irq/1427-nvidia] root 5110 0.0 0.0 0 0 ? S Jun20 0:00 [irq/1428-nvidia] root 5111 0.0 0.0 0 0 ? S Jun20 0:00 [irq/1429-nvidia] root 5112 0.0 0.0 0 0 ? S Jun20 0:00 [irq/1430-nvidia] root 5113 0.0 0.0 0 0 ? S Jun20 0:00 [irq/1431-nvidia] root 5114 0.0 0.0 0 0 ? S Jun20 0:00 [irq/1432-nvidia] root 5115 0.0 0.0 0 0 ? S Jun20 0:00 [irq/1433-nvidia] root 5116 0.0 0.0 0 0 ? S Jun20 0:00 [irq/1434-nvidia] root 5117 0.0 0.0 0 0 ? S Jun20 0:00 [nvidia] nobody 6200 2.0 0.0 1243440 27116 ? Ssl Jun20 2971:48 /bin/node_exporter --path.procfs=/host/proc --path.sysfs=/host/sys --path.rootfs=/host/root --web.listen-address=[0.0.0.0]:9100 --no-collector.arp --no-collector.ipvs --no-collector.softnet --collector.filesystem.mount-points-exclude=^/(dev|proc|sys|var/lib/docker/.+|var/lib/kubelet/.+|mnt/op|mnt/airs-business|tmp/snap.+|run/containerd/.+)($|/) --collector.filesystem.fs-types-exclude=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ --collector.textfile.directory=/run/node-exporter --collector.systemd --collector.systemd.unit-include=(containerd|kubelet|nvidia-fabricmanager|rsyslog|leofs_mond|ssh).service root 11377 0.0 0.0 3098828 33376 ? Ssl Jun20 16:08 nvidia-device-plugin 5) Any outbound TCP to common mining ports? n/a