For some reason, tensorflow is not using GPU even it’s showing in the device list, and we rebooted the Lambda. Then:
nvidia-smi has failed because it couldn’t communicate with the NVIDIA driver.
Anyway to fix it?
output of dpkg -l |grep nvidia
ii libnvidia-cfg1-450:amd64 450.102.04-0ubuntu0.18.04.1 amd64 NVIDIA binary OpenGL/GLX configuration library
ii libnvidia-common-450 450.102.04-0ubuntu0.18.04.1 all Shared files used by the NVIDIA libraries
ii libnvidia-compute-450:amd64 450.102.04-0ubuntu0.18.04.1 amd64 NVIDIA libcompute package
ii libnvidia-compute-450:i386 450.102.04-0ubuntu0.18.04.1 i386 NVIDIA libcompute package
ii libnvidia-decode-450:amd64 450.102.04-0ubuntu0.18.04.1 amd64 NVIDIA Video Decoding runtime libraries
ii libnvidia-decode-450:i386 450.102.04-0ubuntu0.18.04.1 i386 NVIDIA Video Decoding runtime libraries
ii libnvidia-encode-450:amd64 450.102.04-0ubuntu0.18.04.1 amd64 NVENC Video Encoding runtime library
ii libnvidia-encode-450:i386 450.102.04-0ubuntu0.18.04.1 i386 NVENC Video Encoding runtime library
ii libnvidia-extra-450:amd64 450.102.04-0ubuntu0.18.04.1 amd64 Extra libraries for the NVIDIA driver
ii libnvidia-fbc1-450:amd64 450.102.04-0ubuntu0.18.04.1 amd64 NVIDIA OpenGL-based Framebuffer Capture runtime library
ii libnvidia-fbc1-450:i386 450.102.04-0ubuntu0.18.04.1 i386 NVIDIA OpenGL-based Framebuffer Capture runtime library
ii libnvidia-gl-450:amd64 450.102.04-0ubuntu0.18.04.1 amd64 NVIDIA OpenGL/GLX/EGL/GLES GLVND libraries and Vulkan ICD
ii libnvidia-gl-450:i386 450.102.04-0ubuntu0.18.04.1 i386 NVIDIA OpenGL/GLX/EGL/GLES GLVND libraries and Vulkan ICD
ii libnvidia-ifr1-450:amd64 450.102.04-0ubuntu0.18.04.1 amd64 NVIDIA OpenGL-based Inband Frame Readback runtime library
ii libnvidia-ifr1-450:i386 450.102.04-0ubuntu0.18.04.1 i386 NVIDIA OpenGL-based Inband Frame Readback runtime library
ii nvidia-compute-utils-450 450.102.04-0ubuntu0.18.04.1 amd64 NVIDIA compute utilities
ii nvidia-cuda-dev:amd64 10.0.130-0lambda3 amd64 CUDA development files
ii nvidia-cuda-doc 10.0.130-0lambda3 all CUDA toolkit documentation
ii nvidia-cuda-gdb 10.0.130-0lambda3 amd64 CUDA Debugger
ii nvidia-cuda-toolkit 10.0.130-0lambda3 amd64 CUDA development toolkit
ii nvidia-dkms-450 450.102.04-0ubuntu0.18.04.1 amd64 NVIDIA DKMS package
ii nvidia-driver-440 450.66-0lambda0~18.04.2 amd64 Transitional package for nvidia-driver-450
ii nvidia-driver-450 450.102.04-0ubuntu0.18.04.1 amd64 NVIDIA driver metapackage
ii nvidia-kernel-common-450 450.102.04-0ubuntu0.18.04.1 amd64 Shared files used with the kernel module
ii nvidia-kernel-source-450 450.102.04-0ubuntu0.18.04.1 amd64 NVIDIA kernel source package
ii nvidia-modprobe 455.45.01-0lambda1 amd64 utility to load NVIDIA kernel modules and create device nodes
ii nvidia-profiler 10.0.130-0lambda3 amd64 NVIDIA CUDA profiler
ii nvidia-settings 455.45.01-0lambda1 amd64 Tool for configuring the NVIDIA graphics driver
ii nvidia-utils-450 450.102.04-0ubuntu0.18.04.1 amd64 NVIDIA driver support binaries
ii xserver-xorg-video-nvidia-450 450.102.04-0ubuntu0.18.04.1 amd64 NVIDIA binary Xorg driver