Simple Ubuntu Auto-update KILLS Nvidia on Tensorbook

Technical Help
1

Running Razer x Lambda Tensorbook 2022 on Ubuntu 22.04.1 LTS. After a silent upgrade, well, nvidia-smi returns:

NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver. Make sure that the latest NVIDIA driver is installed and running.

I ran:

sudo rm -f /etc/apt/sources.list.d/{graphics,nvidia,cuda}* && \
dpkg -l | \
awk '/cuda|lib(accinj64|cu(blas|dart|dnn|fft|inj|pti|rand|solver|sparse)|magma|nccl|npp|nv[^p])|nv(idia|ml)|tensor(flow|board)|torch/ { print $2 }' | \
sudo xargs -or apt -y remove --purge
sudo apt -y update && sudo apt -y install lambda-stack-cuda lambda-tensorbook

per the instructions here How do I reinstall Lambda Stack in Ubuntu on my Tensorbook? | Lambda Docs, and am getting stuck with some kind of driver error.

Starting around here:

DEBUG:Parsing /usr/share/ubuntu-drivers-common/quirks/dell_latitude
Loading new nvidia-550.90.07 DKMS files...
Building for 6.5.0-35-generic
Building for architecture x86_64
Building initial module for 6.5.0-35-generic
ERROR: Cannot create report: [Errno 17] File exists: '/var/crash/nvidia-dkms-550
.0.crash'
Error! Bad return status for module build on kernel: 6.5.0-35-generic (x86_64)
Consult /var/lib/dkms/nvidia/550.90.07/build/make.log for more information.
dpkg: error processing package nvidia-dkms-550 (--configure):
 installed nvidia-dkms-550 package post-installation script subprocess returned 
error exit status 10

followed by a host of dependency errors:

Setting up python3-torch-cuda (2.3.1+ds-0lambda0.22.04.1) ...
dpkg: dependency problems prevent configuration of nvidia-driver-550:
 nvidia-driver-550 depends on nvidia-dkms-550 (= 550.90.07-0lambda0.22.04.2); ho
wever:
  Package nvidia-dkms-550 is not configured yet.

dpkg: error processing package nvidia-driver-550 (--configure):
 dependency problems - leaving unconfigured
Setting up nvidia-cuda-dev:amd64 (12.2.140~12.2.2-0lambda0.22.04.1) ...
No apport report written because the error message indicates its a followup erro
r from a previous failure.
                          Setting up libxext6:i386 (2:1.3.4-1build1) ...
Setting up libnvidia-fbc1-550:i386 (550.90.07-0lambda0.22.04.2) ...
Setting up nvidia-cuda-toolkit (12.2.140~12.2.2-0lambda0.22.04.1) ...
Setting up nvidia-opencl-dev:amd64 (12.2.140~12.2.2-0lambda0.22.04.1) ...
dpkg: dependency problems prevent configuration of nvidia-driver-535:
 nvidia-driver-535 depends on nvidia-driver-550; however:
  Package nvidia-driver-550 is not configured yet.

dpkg: error processing package nvidia-driver-535 (--configure):
 dependency problems - leaving unconfigured
Setting up libnvidia-gl-550:i386 (550.90.07-0lambda0.22.04.2) ...
No apport report written because the error message indicates its a followup erro
r from a previous failure.
                          Setting up python3-torchvision-cuda (0.18.1-0lambda0.2
2.04.1) ...
Setting up libnvidia-decode-550:i386 (550.90.07-0lambda0.22.04.2) ...
Setting up python3-jaxlib-cuda (0.4.14-0lambda0.22.04.2) ...
Setting up libnvidia-encode-550:i386 (550.90.07-0lambda0.22.04.2) ...
Setting up python3-triton-cuda (2.3.1+llvm18.1.6-0lambda0.22.04.1) ...
dpkg: dependency problems prevent configuration of nvidia-driver-525:
 nvidia-driver-525 depends on nvidia-driver-535; however:
  Package nvidia-driver-535 is not configured yet.

dpkg: error processing package nvidia-driver-525 (--configure):
 dependency problems - leaving unconfigured
Setting up python3-jax-cuda (0.4.14-0lambda0.22.04.1) ...
No apport report written because MaxReports is reached already
                                                              Setting up python3
-tensorflow-cuda (2.13.1-0lambda0.22.04.2) ...

Further. the link to the recovery image here is broken lol: Where can I download recovery images? | Lambda Docs.

Is the Tensorbook like a completely dead product? Seems like no official support, all the docs in beta…

3

Logfile issues start here:

# CC [M]  /var/lib/dkms/nvidia/550.90.07/build/nvidia/nv-cray.o
  cc -Wp,-MMD,/var/lib/dkms/nvidia/550.90.07/build/nvidia/.nv-cray.o.d -nostdinc -I./arch/x86/include -I./arch/x86/include/generated  -I./include -I./arch/x86/include/uapi -I./arch/x86/include/generated/uapi -I./include/uapi -I./include/generated/uapi -include ./include/linux/compiler-version.h -include ./include/linux/kconfig.h -I./ubuntu/include -include ./include/linux/compiler_types.h -D__KERNEL__ -std=gnu11 -fshort-wchar -funsigned-char -fno-common -fno-PIE -fno-strict-aliasing -Wall -Wundef -Werror=implicit-function-declaration -Werror=implicit-int -Werror=return-type -Werror=strict-prototypes -Wno-format-security -Wno-trigraphs -mno-sse -mno-mmx -mno-sse2 -mno-3dnow -mno-avx -fcf-protection=none -m64 -falign-jumps=1 -falign-loops=1 -mno-80387 -mno-fp-ret-in-387 -mpreferred-stack-boundary=3 -mskip-rax-setup -mtune=generic -mno-red-zone -mcmodel=kernel -Wno-sign-compare -fno-asynchronous-unwind-tables -mindirect-branch=thunk-extern -mindirect-branch-register -mindirect-branch-cs-prefix -mfunction-return=thunk-extern -fno-jump-tables -mharden-sls=all -fpatchable-function-entry=16,16 -fno-delete-null-pointer-checks -Wno-frame-address -Wno-format-truncation -Wno-format-overflow -Wno-address-of-packed-member -O2 -fno-allow-store-data-races -Wframe-larger-than=1024 -fstack-protector-strong -Wno-main -Wno-unused-but-set-variable -Wno-unused-const-variable -fno-omit-frame-pointer -fno-optimize-sibling-calls -ftrivial-auto-var-init=zero -fzero-call-used-regs=used-gpr -pg -mrecord-mcount -falign-functions=16 -Wvla -Wno-pointer-sign -Wno-maybe-uninitialized -Wno-array-bounds -Wno-alloc-size-larger-than -Wimplicit-fallthrough=5 -fno-strict-overflow -fno-stack-check -fconserve-stack -Werror=date-time -g -gdwarf-5 -Wno-error -I/var/lib/dkms/nvidia/550.90.07/build/common/inc -I/var/lib/dkms/nvidia/550.90.07/build -Wall -Wno-cast-qual -Wno-format-extra-args -D__KERNEL__ -DMODULE -DNVRM -DNV_VERSION_STRING=\"550.90.07\" -DNV_FILESYSTEM_ACCESS_AVAILABLE=1 -Wno-unused-function -Wuninitialized -fno-strict-aliasing -ffreestanding -mno-red-zone -mcmodel=kernel -DNV_UVM_ENABLE -DNV_SPECTRE_V2=0 -DNV_KERNEL_INTERFACE_LAYER -I/var/lib/dkms/nvidia/550.90.07/build/nvidia -DNVIDIA_UNDEF_LEGACY_BIT_MACROS -UDEBUG -U_DEBUG -DNDEBUG  -fsanitize=bounds-strict -fsanitize=shift -fsanitize=bool -fsanitize=enum  -DMODULE  -DKBUILD_BASENAME='"nv_cray"' -DKBUILD_MODNAME='"nvidia"' -D__KBUILD_MODNAME=kmod_nvidia -c -o /var/lib/dkms/nvidia/550.90.07/build/nvidia/nv-cray.o /var/lib/dkms/nvidia/550.90.07/build/nvidia/nv-cray.c   ; ./tools/objtool/objtool --hacks=jump_label --hacks=noinstr --hacks=skylake --retpoline --rethunk --sls --stackval --static-call --uaccess --prefix=16   --module /var/lib/dkms/nvidia/550.90.07/build/nvidia/nv-cray.o
cc: error: unrecognized command-line option ‘-ftrivial-auto-var-init=zero’
# CC [M]  /var/lib/dkms/nvidia/550.90.07/build/nvidia/nv-dma.o
  cc -Wp,-MMD,/var/lib/dkms/nvidia/550.90.07/build/nvidia/.nv-dma.o.d -nostdinc -I./arch/x86/include -I./arch/x86/include/generated  -I./include -I./arch/x86/include/uapi -I./arch/x86/include/generated/uapi -I./include/uapi -I./include/generated/uapi -include ./include/linux/compiler-version.h -include ./include/linux/kconfig.h -I./ubuntu/include -include ./include/linux/compiler_types.h -D__KERNEL__ -std=gnu11 -fshort-wchar -funsigned-char -fno-common -fno-PIE -fno-strict-aliasing -Wall -Wundef -Werror=implicit-function-declaration -Werror=implicit-int -Werror=return-type -Werror=strict-prototypes -Wno-format-security -Wno-trigraphs -mno-sse -mno-mmx -mno-sse2 -mno-3dnow -mno-avx -fcf-protection=none -m64 -falign-jumps=1 -falign-loops=1 -mno-80387 -mno-fp-ret-in-387 -mpreferred-stack-boundary=3 -mskip-rax-setup -mtune=generic -mno-red-zone -mcmodel=kernel -Wno-sign-compare -fno-asynchronous-unwind-tables -mindirect-branch=thunk-extern -mindirect-branch-register -mindirect-branch-cs-prefix -mfunction-return=thunk-extern -fno-jump-tables -mharden-sls=all -fpatchable-function-entry=16,16 -fno-delete-null-pointer-checks -Wno-frame-address -Wno-format-truncation -Wno-format-overflow -Wno-address-of-packed-member -O2 -fno-allow-store-data-races -Wframe-larger-than=1024 -fstack-protector-strong -Wno-main -Wno-unused-but-set-variable -Wno-unused-const-variable -fno-omit-frame-pointer -fno-optimize-sibling-calls -ftrivial-auto-var-init=zero -fzero-call-used-regs=used-gpr -pg -mrecord-mcount -falign-functions=16 -Wvla -Wno-pointer-sign -Wno-maybe-uninitialized -Wno-array-bounds -Wno-alloc-size-larger-than -Wimplicit-fallthrough=5 -fno-strict-overflow -fno-stack-check -fconserve-stack -Werror=date-time -g -gdwarf-5 -Wno-error -I/var/lib/dkms/nvidia/550.90.07/build/common/inc -I/var/lib/dkms/nvidia/550.90.07/build -Wall -Wno-cast-qual -Wno-format-extra-args -D__KERNEL__ -DMODULE -DNVRM -DNV_VERSION_STRING=\"550.90.07\" -DNV_FILESYSTEM_ACCESS_AVAILABLE=1 -Wno-unused-function -Wuninitialized -fno-strict-aliasing -ffreestanding -mno-red-zone -mcmodel=kernel -DNV_UVM_ENABLE -DNV_SPECTRE_V2=0 -DNV_KERNEL_INTERFACE_LAYER -I/var/lib/dkms/nvidia/550.90.07/build/nvidia -DNVIDIA_UNDEF_LEGACY_BIT_MACROS -UDEBUG -U_DEBUG -DNDEBUG  -fsanitize=bounds-strict -fsanitize=shift -fsanitize=bool -fsanitize=enum  -DMODULE  -DKBUILD_BASENAME='"nv_dma"' -DKBUILD_MODNAME='"nvidia"' -D__KBUILD_MODNAME=kmod_nvidia -c -o /var/lib/dkms/nvidia/550.90.07/build/nvidia/nv-dma.o /var/lib/dkms/nvidia/550.90.07/build/nvidia/nv-dma.c   ; ./tools/objtool/objtool --hacks=jump_label --hacks=noinstr --hacks=skylake --retpoline --rethunk --sls --stackval --static-call --uaccess --prefix=16   --module /var/lib/dkms/nvidia/550.90.07/build/nvidia/nv-dma.o
make[3]: *** [scripts/Makefile.build:251: /var/lib/dkms/nvidia/550.90.07/build/nvidia/nv.o] Error 1
make[3]: *** Waiting for unfinished jobs....
cc: error: unrecognized command-line option ‘-ftrivial-auto-var-init=zero’
make[3]: *** [scripts/Makefile.build:251: /var/lib/dkms/nvidia/550.90.07/build/nvidia/nv-pci.o] Error 1
cc: error: unrecognized command-line option ‘-ftrivial-auto-var-init=zero’
make[3]: *** [scripts/Makefile.build:251: /var/lib/dkms/nvidia/550.90.07/build/nvidia/nv-dmabuf.o] Error 1
cc: error: unrecognized command-line option ‘-ftrivial-auto-var-init=zero’
make[3]: *** [scripts/Makefile.build:251: /var/lib/dkms/nvidia/550.90.07/build/nvidia/nv-nano-timer.o] Error 1
cc: error: unrecognized command-line option ‘-ftrivial-auto-var-init=zero’
make[3]: *** [scripts/Makefile.build:251: /var/lib/dkms/nvidia/550.90.07/build/nvidia/nv-acpi.o] Error 1
cc: error: unrecognized command-line option ‘-ftrivial-auto-var-init=zero’
make[3]: *** [scripts/Makefile.build:251: /var/lib/dkms/nvidia/550.90.07/build/nvidia/nv-cray.o] Error 1
cc: error: unrecognized command-line option ‘-ftrivial-auto-var-init=zero’
make[3]: *** [scripts/Makefile.build:251: /var/lib/dkms/nvidia/550.90.07/build/nvidia/nv-dma.o] Error 1
make[2]: *** [/usr/src/linux-headers-6.5.0-35-generic/Makefile:2039: /var/lib/dkms/nvidia/550.90.07/build] Error 2
make[1]: *** [Makefile:234: __sub-make] Error 2
make[1]: Leaving directory '/usr/src/linux-headers-6.5.0-35-generic'
make: *** [Makefile:85: modules] Error 2
4

Further. the link to the recovery image here is broken lol: Where can I download recovery images? | Lambda Docs.

The recovery image can be downloaded from Lambda Stack and recovery images | Lambda Docs.