How to install Nvidia driver 525 on a Lambda instance

Technical Help
1

I need to install the 525 driver to run the Hailo Dataflow Compiler to compile a neural network. I already did this a month ago and I believe then it worked on the A6000 GPU with a simple:
sudo apt-get remove --purge ‘^nvidia-.*’
sudo apt-get autoremove
sudo apt-get autoclean
sudo ubuntu-drivers install nvidia-driver:525
But now the 525 driver isn’t available in the: ´sudo ubuntu-drivers list´.
I tried installing with ´sudo apt get install nvidia-driver-525´ but this just installed the 550 driver again.
I also tried installing it with dpkg and the file ´nvidia-driver-local-repo-ubuntu2204-525.125.06_1.0-1_amd64.deb´, but then there was no nvidia-smi and after installing nvidia-smi it said the driver was version 550.
Then I tried removing the Lambda specific urls from ´/etc/apt/sources.list.d/´ and then it installed driver 535 instead of 525.
Then I figured I needed to install another linux kernel, version 5.19.0, but after changing the DEFAULT_MODE variable in the /etc/default/grub file and running ´sudo update-grub´ but after rebooting it kept booting in kernel 6.27.0.

What could I do to install the 525 driver instead of the 550 driver on a Lambda instance? I’m new to machine learning and working with GPUs so I don’t really know what I’m doing, some usefull pointers would be much appreciated!

Thanks in advance

Rik

2

I think I am facing a similar issue as you. I have tried installing the drivers with lambda stack as well, but the 535 version. I am using A5000 which have an issue on the 550 version, hence the 535 drivers.

I ran apt search nvidia-driver-535 and I got something interesting:

nvidia-driver-525/unknown 535.183.01-0lambda0.22.04.1 amd64
  Transitional package for nvidia-driver-535
nvidia-driver-535/unknown 550.107.02-0lambda0.22.04.2 amd64
  Transitional package for nvidia-driver-550

Do you see it? Let me run apt search nvidia-driver-525:

nvidia-driver-520/unknown 525.125.06-0lambda0.22.04.1 amd64
  Transitional package for nvidia-driver-525
nvidia-driver-525/unknown 535.183.01-0lambda0.22.04.1 amd64
  Transitional package for nvidia-driver-535

There is an off by one error. Somehow the 525 drivers are tagged as 520, 535 as 525, 550 as 535. However, even if you install the version-1, you end up with 550 drivers.