Under the lastest Jetpack version 4.2.1 for embedded devices like Xavier or TX2 I'm seeing this issue when trying to cmake:

cmake .. -DNVML_RETRIEVE_HEADER_ONLINE=True

I get the error:

-- The C compiler identification is GNU 7.4.0
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
CMake Error at /usr/local/share/cmake-3.15/Modules/FindPackageHandleStandardArgs.cmake:137 (message):
  Could NOT find NVML (missing: NVML_LIBRARIES) (found version "10")
Call Stack (most recent call first):
  /usr/local/share/cmake-3.15/Modules/FindPackageHandleStandardArgs.cmake:378 (_FPHSA_FAILURE_MESSAGE)
  cmake/modules/FindNVML.cmake:52 (FIND_PACKAGE_HANDLE_STANDARD_ARGS)
  CMakeLists.txt:31 (find_package)


-- Configuring incomplete, errors occurred!
See also "/home/nvidia/Documents/nvtop/build/CMakeFiles/CMakeOutput.log".

It seems it finds NVML and doesn't find it in the download part where you look for:

https://raw.githubusercontent.com/NVIDIA/nvidia-settings/master/src/nvml.h

Any insight is appreciated!

Hi, I'm installing nvtop on Ubuntu 18.04.5 LTS following the build instruction in this repo. The build went smooth and there are no warning or error.

But when trying to launch, nvtop, I got the error: Segmentation fault (core dumped)

Here is my nvidia-smi output:

Mon May 24 04:28:38 2021
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 465.19.01    Driver Version: 460.32.03    CUDA Version: 11.2     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  Tesla P100-PCIE...  Off  | 00000000:00:04.0 Off |                    0 |
| N/A   40C    P0    39W / 250W |   3025MiB / 16280MiB |     19%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
+-----------------------------------------------------------------------------+

Also I did some test and find out that the last working commit for me was 0ef51c99ee1598cfc7fd806f122fbaf315f81a26. From that point on I always get this error.

Are there anything I can help to resolve this. Thanks ;)

Hi,

thank you for your great library. It gives a very good overview. One question: Is it possible to store the sorting order similar to htop (which creates a small config file usually at $HOME/.config/htop/htoprc when exiting the application). Then on restart, it loads the setting and applies it.

Is this already implemented and I didn't find it, or is it a kind of new feature?

Best regards Sven

Hi, found this issue today.

The GPU usage jumps to huge % number (screenshot below).

My System: Ubuntu 18.04.5 LTS nvtop version 1.2.0

Please let me know what additional info would be helpfull.

P.$ Thanks for the tool!

Running driver version 450.57 on Ubuntu 18.04.

Running CMake succeeds, however make fails with the following output:

[100%] Linking C executable nvtop
CMakeFiles/nvtop.dir/extract_gpuinfo.c.o: In function `update_device_infos':
extract_gpuinfo.c:(.text+0x584): undefined reference to `nvmlDeviceGetGraphicsRunningProcesses_v2'
extract_gpuinfo.c:(.text+0x624): undefined reference to `nvmlDeviceGetComputeRunningProcesses_v2'
clang: error: linker command failed with exit code 1 (use -v to see invocation)
src/CMakeFiles/nvtop.dir/build.make:253: recipe for target 'src/nvtop' failed
make[2]: *** [src/nvtop] Error 1
CMakeFiles/Makefile2:117: recipe for target 'src/CMakeFiles/nvtop.dir/all' failed
make[1]: *** [src/CMakeFiles/nvtop.dir/all] Error 2
Makefile:129: recipe for target 'all' failed
make: *** [all] Error 2

Anyone had this error?

I've verified that I can access the GPU using nvidia-smi, as well as running Tensorflow:

>>> from tensorflow.python.client import device_lib
>>> print(device_lib.list_local_devices())

Hey there - thanks for this amazing monitoring tool! :bow:

Here's an issue I'm hitting: when running on a 6-gpu system nvtop allocates memory until it is getting killed by the linux oom killer. It looks like there is an overflow somewhere leading to unbound memory allocation (at a rate of multiple GBs per second).

Another data point: this behavior stops to happen when I run in a small terminal (e.g. 80x24) or in a tmux split pane, which indicates it has something to do with the live utilization plots.

When running a debug build and sending a SIGHUP signal during the memory allocation I get backtraces indicating draw_plots in the problem, e.g.

(gdb) bt
#0  0x00005555555c11c8 in nvtop_line_plot (win=0x0, num_data=3200171704, data=0x7ff9f9354800, min=0, max=100, num_plots=4, legend=0x7fffffffe000) at /home/djh/nvtop/src/plot.c:61
#1  0x00005555555a99db in draw_plots (interface=0x6110000002c0) at /home/djh/nvtop/src/interface.c:1604
#2  0x00005555555a9ce1 in draw_gpu_info_ncurses (dev_info=0x61a000000c80, interface=0x6110000002c0) at /home/djh/nvtop/src/interface.c:1625
#3  0x0000555555594192 in main (argc=1, argv=0x7fffffffe4a8) at /home/djh/nvtop/src/nvtop.c:270

Hope that helps, let me know if you need more information.

I manually build nvtop in my Ubuntu 20.04 docker container, becuase the driver problem in the pinned issue. However, I found the pid it's showed it's wrong, because I cannot find the pid using ps:

find pid in ps -aux

A picture is worth a thousand words:

This PR isn't fully ready for merge yet. I think I'd like some feedback and do some polishing first (hence the RFC). My AMDGPU is also an iGPU / APU which limits how much I can test myself.

Most the access is using libdrm and kernel APIs. <drm/amdgpu_drm.h> is a kernel UAPI header so I directly included it. For libdrm I'm using dlopen and re-declaring the constants to avoid a compile-time dependency, just like what the original code does with NVML.

I also did some code refactoring to make my life easier, hope that is okay. ~~I really cannot deal with typedef structs~~

CC #106, i915 kernel changes doesn't seem to have merged into mainline yet.

Distributor ID: Ubuntu Description: Ubuntu 16.04.7 LTS Release: 16.04 Codename: xenial

I tried to build nvtop from source but got these error messages

CMakeFiles/nvtop.dir/extract_gpuinfo_amdgpu.c.o: In function `gpuinfo_amdgpu_get_running_processes':
extract_gpuinfo_amdgpu.c:(.text+0x1232): undefined reference to `reallocarray'
extract_gpuinfo_amdgpu.c:(.text+0x12d8): undefined reference to `reallocarray'
collect2: error: ld returned 1 exit status
src/CMakeFiles/nvtop.dir/build.make:290: recipe for target 'src/nvtop' failed
make[2]: *** [src/nvtop] Error 1
CMakeFiles/Makefile2:124: recipe for target 'src/CMakeFiles/nvtop.dir/all' failed
make[1]: *** [src/CMakeFiles/nvtop.dir/all] Error 2
Makefile:135: recipe for target 'all' failed
make: *** [all] Error 2

I searched for this reallocarray function, it is just a Standard C Library(libc, -lc) I don't think it can be missing. So I think it is because of lib link problem

Here below is the makefile generated by cmake:

# CMAKE generated file: DO NOT EDIT!
# Generated by "Unix Makefiles" Generator, CMake Version 3.23

# Default target executed when no arguments are given to make.
default_target: all
.PHONY : default_target

# Allow only one "make -f Makefile2" at a time, but pass parallelism.
.NOTPARALLEL:

#=============================================================================
# Special targets provided by cmake.

# Disable implicit rules so canonical targets will work.
.SUFFIXES:

# Disable VCS-based implicit rules.
% : %,v

# Disable VCS-based implicit rules.
% : RCS/%

# Disable VCS-based implicit rules.
% : RCS/%,v

# Disable VCS-based implicit rules.
% : SCCS/s.%

# Disable VCS-based implicit rules.
% : s.%

.SUFFIXES: .hpux_make_needs_suffix_list

# Command-line flag to silence nested $(MAKE).
$(VERBOSE)MAKESILENT = -s

#Suppress display of executed commands.
$(VERBOSE).SILENT:

# A target that is always out of date.
cmake_force:
.PHONY : cmake_force

#=============================================================================
# Set environment variables for the build.

# The shell in which to execute make rules.
SHELL = /bin/sh

# The CMake executable.
CMAKE_COMMAND = /opt/cmake/bin/cmake

# The command to remove a file.
RM = /opt/cmake/bin/cmake -E rm -f

# Escaping for special characters.
EQUALS = =

# The top-level source directory on which CMake was run.
CMAKE_SOURCE_DIR = /home/amoschenyq/nvtop

# The top-level build directory on which CMake was run.
CMAKE_BINARY_DIR = /home/amoschenyq/nvtop/build

#=============================================================================
# Targets provided globally by CMake.

# Special rule for the target edit_cache
edit_cache:
	@$(CMAKE_COMMAND) -E cmake_echo_color --switch=$(COLOR) --cyan "Running CMake cache editor..."
	/opt/cmake/bin/ccmake -S$(CMAKE_SOURCE_DIR) -B$(CMAKE_BINARY_DIR)
.PHONY : edit_cache

# Special rule for the target edit_cache
edit_cache/fast: edit_cache
.PHONY : edit_cache/fast

# Special rule for the target rebuild_cache
rebuild_cache:
	@$(CMAKE_COMMAND) -E cmake_echo_color --switch=$(COLOR) --cyan "Running CMake to regenerate build system..."
	/opt/cmake/bin/cmake --regenerate-during-build -S$(CMAKE_SOURCE_DIR) -B$(CMAKE_BINARY_DIR)
.PHONY : rebuild_cache

# Special rule for the target rebuild_cache
rebuild_cache/fast: rebuild_cache
.PHONY : rebuild_cache/fast

# Special rule for the target list_install_components
list_install_components:
	@$(CMAKE_COMMAND) -E cmake_echo_color --switch=$(COLOR) --cyan "Available install components are: \"Unspecified\""
.PHONY : list_install_components

# Special rule for the target list_install_components
list_install_components/fast: list_install_components
.PHONY : list_install_components/fast

# Special rule for the target install
install: preinstall
	@$(CMAKE_COMMAND) -E cmake_echo_color --switch=$(COLOR) --cyan "Install the project..."
	/opt/cmake/bin/cmake -P cmake_install.cmake
.PHONY : install

# Special rule for the target install
install/fast: preinstall/fast
	@$(CMAKE_COMMAND) -E cmake_echo_color --switch=$(COLOR) --cyan "Install the project..."
	/opt/cmake/bin/cmake -P cmake_install.cmake
.PHONY : install/fast

# Special rule for the target install/local
install/local: preinstall
	@$(CMAKE_COMMAND) -E cmake_echo_color --switch=$(COLOR) --cyan "Installing only the local directory..."
	/opt/cmake/bin/cmake -DCMAKE_INSTALL_LOCAL_ONLY=1 -P cmake_install.cmake
.PHONY : install/local

# Special rule for the target install/local
install/local/fast: preinstall/fast
	@$(CMAKE_COMMAND) -E cmake_echo_color --switch=$(COLOR) --cyan "Installing only the local directory..."
	/opt/cmake/bin/cmake -DCMAKE_INSTALL_LOCAL_ONLY=1 -P cmake_install.cmake
.PHONY : install/local/fast

# Special rule for the target install/strip
install/strip: preinstall
	@$(CMAKE_COMMAND) -E cmake_echo_color --switch=$(COLOR) --cyan "Installing the project stripped..."
	/opt/cmake/bin/cmake -DCMAKE_INSTALL_DO_STRIP=1 -P cmake_install.cmake
.PHONY : install/strip

# Special rule for the target install/strip
install/strip/fast: preinstall/fast
	@$(CMAKE_COMMAND) -E cmake_echo_color --switch=$(COLOR) --cyan "Installing the project stripped..."
	/opt/cmake/bin/cmake -DCMAKE_INSTALL_DO_STRIP=1 -P cmake_install.cmake
.PHONY : install/strip/fast

# The main all target
all: cmake_check_build_system
	$(CMAKE_COMMAND) -E cmake_progress_start /home/amoschenyq/nvtop/build/CMakeFiles /home/amoschenyq/nvtop/build//CMakeFiles/progress.marks
	$(MAKE) $(MAKESILENT) -f CMakeFiles/Makefile2 all
	$(CMAKE_COMMAND) -E cmake_progress_start /home/amoschenyq/nvtop/build/CMakeFiles 0
.PHONY : all

# The main clean target
clean:
	$(MAKE) $(MAKESILENT) -f CMakeFiles/Makefile2 clean
.PHONY : clean

# The main clean target
clean/fast: clean
.PHONY : clean/fast

# Prepare targets for installation.
preinstall: all
	$(MAKE) $(MAKESILENT) -f CMakeFiles/Makefile2 preinstall
.PHONY : preinstall

# Prepare targets for installation.
preinstall/fast:
	$(MAKE) $(MAKESILENT) -f CMakeFiles/Makefile2 preinstall
.PHONY : preinstall/fast

# clear depends
depend:
	$(CMAKE_COMMAND) -S$(CMAKE_SOURCE_DIR) -B$(CMAKE_BINARY_DIR) --check-build-system CMakeFiles/Makefile.cmake 1
.PHONY : depend

#=============================================================================
# Target rules for targets named uninstall

# Build rule for target.
uninstall: cmake_check_build_system
	$(MAKE) $(MAKESILENT) -f CMakeFiles/Makefile2 uninstall
.PHONY : uninstall

# fast build rule for target.
uninstall/fast:
	$(MAKE) $(MAKESILENT) -f CMakeFiles/uninstall.dir/build.make CMakeFiles/uninstall.dir/build
.PHONY : uninstall/fast

#=============================================================================
# Target rules for targets named nvtop

# Build rule for target.
nvtop: cmake_check_build_system
	$(MAKE) $(MAKESILENT) -f CMakeFiles/Makefile2 nvtop
.PHONY : nvtop

# fast build rule for target.
nvtop/fast:
	$(MAKE) $(MAKESILENT) -f src/CMakeFiles/nvtop.dir/build.make src/CMakeFiles/nvtop.dir/build
.PHONY : nvtop/fast

# Help Target
help:
	@echo "The following are some of the valid targets for this Makefile:"
	@echo "... all (the default if no target is provided)"
	@echo "... clean"
	@echo "... depend"
	@echo "... edit_cache"
	@echo "... install"
	@echo "... install/local"
	@echo "... install/strip"
	@echo "... list_install_components"
	@echo "... rebuild_cache"
	@echo "... uninstall"
	@echo "... nvtop"
.PHONY : help



#=============================================================================
# Special targets to cleanup operation of make.

# Special rule to run CMake to check the build system integrity.
# No rule that depends on this can have commands that come from listfiles
# because they might be regenerated.
cmake_check_build_system:
	$(CMAKE_COMMAND) -S$(CMAKE_SOURCE_DIR) -B$(CMAKE_BINARY_DIR) --check-build-system CMakeFiles/Makefile.cmake 0
.PHONY : cmake_check_build_system

Dear Syllo,

I created a very simple python script to allocate some GPU memory - the corresponding process appeared in nvtop: However, the PID I can see using htop differs from that one:

Do you happen to have any idea? Please note, that this can be a driver related issue as in the output of nvidia-smi there are no processes shown at all. I tried to find some answers for this one as well, but the only thing I found is that in containers it won't work. However, I do not use any containers and I don't have root access either, hence driver updates are not really possible.

Thanks in advance for any advice, Daniel

Hi,

first thanks for this great tool. It really simplifies to keep track of the running GPU processes on our servers. I would also like to also see the CPU utilization. If I would create a pull request, would you welcome this feature?

Thank you, Leon

Hi, I am running nvtop 3.0.1 (installed via apt) on Debian testing. kernel: 6.0.12 cpu: Ryzen 9 7950x gpu0: AMD iGPU gpu1: Nvidia GTX 1080 nvidia-driver: 510.108.03-1 installed from Debian testing non-free repo

When using the menu to select only one gpu for monitoring it selects gpu0 regardless of the menu selection. When I go back to the menu and select the other gpu it switches to gpu1 but immediately crashes with

nvtop: ./src/extract_gpuinfo_amdgpu.c:946: parse_drm_fdinfo_amd: Assertion `!cache_entry_check `&& "We should not be processing a client id twice per update"' failed.

The monitoring for gpu1 works correctly while both gpus are selected at once. If you need any additional system info/logs please let me know.

From dmesg | grep nvtop:

[    4.168334] systemd[1]: Mounting Mount unit for nvtop, revision 66...
[    5.639143] audit: type=1400 audit(1670626277.332:29): apparmor="STATUS" operation="profile_load" profile="unconfined" name="snap-update-ns.nvtop" pid=617 comm="apparmor_parser"
[  108.023408] audit: type=1400 audit(1670626379.550:79): apparmor="DENIED" operation="open" profile="snap.nvtop.nvtop" name="/proc/driver/nvidia/capabilities/mig/config" pid=3823 comm="nvtop" requested_mask="r" denied_mask="r" fsuid=1000 ouid=0
[  108.024424] audit: type=1400 audit(1670626379.550:80): apparmor="DENIED" operation="open" profile="snap.nvtop.nvtop" name="/proc/driver/nvidia/capabilities/mig/config" pid=3823 comm="nvtop" requested_mask="r" denied_mask="r" fsuid=1000 ouid=0
[  108.024426] audit: type=1400 audit(1670626379.550:81): apparmor="DENIED" operation="open" profile="snap.nvtop.nvtop" name="/proc/driver/nvidia/capabilities/mig/config" pid=3823 comm="nvtop" requested_mask="r" denied_mask="r" fsuid=1000 ouid=0
[  108.024427] audit: type=1400 audit(1670626379.550:82): apparmor="DENIED" operation="open" profile="snap.nvtop.nvtop" name="/proc/driver/nvidia/capabilities/mig/monitor" pid=3823 comm="nvtop" requested_mask="r" denied_mask="r" fsuid=1000 ouid=0
[  108.024428] audit: type=1400 audit(1670626379.550:83): apparmor="DENIED" operation="open" profile="snap.nvtop.nvtop" name="/proc/driver/nvidia/capabilities/mig/monitor" pid=3823 comm="nvtop" requested_mask="r" denied_mask="r" fsuid=1000 ouid=0
[  108.024429] audit: type=1400 audit(1670626379.550:84): apparmor="DENIED" operation="open" profile="snap.nvtop.nvtop" name="/proc/driver/nvidia/capabilities/mig/monitor" pid=3823 comm="nvtop" requested_mask="r" denied_mask="r" fsuid=1000 ouid=0
[  110.417257] audit: type=1400 audit(1670626381.942:85): apparmor="DENIED" operation="open" profile="snap.nvtop.nvtop" name="/proc/2040/fdinfo/" pid=3823 comm="nvtop" requested_mask="r" denied_mask="r" fsuid=1000 ouid=1000
[  110.417267] audit: type=1400 audit(1670626381.942:86): apparmor="DENIED" operation="open" profile="snap.nvtop.nvtop" name="/proc/2073/fdinfo/" pid=3823 comm="nvtop" requested_mask="r" denied_mask="r" fsuid=1000 ouid=1000
[  110.417271] audit: type=1400 audit(1670626381.942:87): apparmor="DENIED" operation="open" profile="snap.nvtop.nvtop" name="/proc/2074/fdinfo/" pid=3823 comm="nvtop" requested_mask="r" denied_mask="r" fsuid=1000 ouid=1000

The /proc/$NNNN/fdinfo lines repeat extensively.

From lsb_release -a:

No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 22.04.1 LTS
Release:        22.04
Codename:       jammy

From uname -a (note, I removed the actual hostname below):

Linux $HOSTNAME 5.15.0-56-generic #62-Ubuntu SMP Tue Nov 22 19:54:14 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

I'm new to using nvtop so I'm not sure what exactly might be breaking. The only problem I've seen so far is that the README.md doc says to do snap connect nvtop:kubernetes-support after the snap installation, but that generates the error:

error: snap "nvtop" has no plug named "kubernetes-support"

Drop down terminals like yakuake and guake typically use F12 key as the default key for showing/hiding terminal. While using nvtop in such drop down terminals, F12 function key can not be used to save config. It would be better if F12 key is changed to some other function key