Pytorch and conda on Lambda Workstation RTX 3090

[nathan.ing]

Hi, I’m managing a Lambda Workstation with RTX 3090’s, and fully up to date Lambda stack. This is a shared system, so what I’d like is to let users install and manage their own conda environments. The basic expectation is that this would work:

$ conda create -n pytorch python=3.9
$ conda activate pytorch
(pytorch) $ conda install -c pytorch pytorch torchaudio torchvision

However, the version of pytorch distributed via conda, and pip, is seemingly incompatible with the RTX 3090. They even print an explicit error mentioning using pytorch with the RTX 3090 that points to the install from source instructions.

So, my question to those of you out there that use conda: how are you installing pytorch?

(I believe that my case is different from the one mentioned here, as I believe that post was in reference to the system-wide lambda stack python / pytorch. I’ve just tested that, and it works as expected on the pytorch/mnist example script.)

[markd]

I do not see cuda-toolkit installed above. Remember Anaconda changes your search paths and ignores the system installed software. So either you would need to install the cuda-toolkit you want or set the PYTHONPATH to see the system installed software.

1. Pytorch has a handy tool to help with the ‘mix-and-match’ depending on your needs:
   https://pytorch.org/get-started/locally/
   conda install pytorch torchvision torchaudio cudatoolkit=11.1 -c pytorch -c nvidia
2. You can have a YAML file to save what you have also. (below is a example yaml).
   conda env create -f ./pytorch-latest.yaml
3. You can set the PYTHONPATH so Anaconda can see the system installed software.
   (NOTE: There may be conflicts if you do not have a package in Anaconda and the only
   version seen is newer than what you intended).
   export PYTHONPATH=/usr/lib/python3/dist-packages:${PYTHONPATH}

— Example pytorch-latest.yaml file -------
name: pytorch-latest

channels:
  - pytorch
  - conda-forge
  - nvidia
  - defaults

dependencies:
  - python=3.9
  - cudatoolkit
  - pytorch
  - torchaudio
  - torchvision
  - pip

[atyshka]

@nathan.ing Did you ever find a solution? I am having the same issue: I need to use conda because the torchaudio pip wheel is incompatible with the system install of pytorch, but the version of pytorch with conda is not compatible with the 3000 series gpus

[markd]

There are other options also:
python venv (virtualenv)
docker
anconda - as you mentioned.

Pytorch has a nice example also, and you can specify the level of CUDA and pytorch.
From: https://pytorch.org/get-started/locally/
For CUDA 11.3, linux, conda, python
conda install pytorch torchvision torchaudio cudatoolkit=11.3 -c pytorch

Or you can use the YAML file like I was mentioning and you can specify the cudatoolkit version also.

Mark

[seneca_wolf]

July 2022 after installing Miniconda as per this link I did:

$ conda create -n pytorch python=3.9
$ conda activate pytorch
$ conda install pytorch torchvision torchaudio cudatoolkit=11.1 -c pytorch -c nvidia

And it looks like it is working. Is there any sort of test script available that I could run to see whether everything is copacetic, as I recently upgraded to Ubuntu 20.04 & I’m not clear on whether all of Lambda Stack is still available and working properly.

Thanks!

[markd]

A simple torch test to make sure you see the GPUs, which version confirmed and which version it was compiled with is:

import torch

print("\nPytorch version: “,torch.version)
print(torch._C._cuda_getCompiledVersion(), “cuda compiled version”)
print(”\ntorch",torch.file)
device = torch.device(‘cuda’ if torch.cuda.is_available() else ‘cpu’)
print("Device name: ", torch.cuda.get_device_name(device))
print("Device properties: ", torch.cuda.get_device_properties(device))
print("Device_count: ", torch.cuda.device_count())

[markd]

anaconda/miniconda would not really be using the lambda stack. It installs its own python and pytorch or other packages. It only uses the nvidia driver.

On other issue is Lambda stacks pytorch/tensorflow have a cudnn installed with them.
Anaconda often installs cudnn, but never sets the path for cuDNN, and this I see more commonly with tensorflow in anaconda.

And to get around that you can either:

1. Install cudnn from NVIDIA’s web site (registration and EULA).
   or
2. For anaconda:
   LD_LIBRARY_PATH=${CONDA_PREFIX}/lib:${LD_LIBRARY_PATH} python your-code.py

[markd]

You can specify the version of cudatoolkit you want, or use another channel with support.
https://pytorch.org/get-started/locally/

docker or python venv are other options. python venv can use the system installed software also by using the ‘–system-site-packages’

Has options of what to install. On 3000 series (or any Ampere GPU) you need CUDA 11.1 or higher support.

conda install pytorch torchvision torchaudio cudatoolkit=11.3 -c pytorch
or
pytorch.org states: NOTE: ‘conda-forge’ channel is required for cudatoolkit 11.6
conda install pytorch torchvision torchaudio cudatoolkit=11.6 -c pytorch -c conda-forge

NVIDIA also has a channel so this would work also:
conda install pytorch torchvision torchaudio cudatoolkit=11.6 -c pytorch -c nvidia

[seneca_wolf]

get_device_name

Thanks so much!

I don’t know if this is too much to look at, but here is a bit of an interactive session I had just now trying some of the above commands. I find that in my newly fabricated conda environment I don’t have access to the GPU. Same thing in system Python3. Only in Python2 do I have access to the GPU.

I’m not sure what to do.

Thank you for all your help!

senecawolf@tensorbook:~/Desktop/test-project$ conda info --envs
# conda environments:
#
base                  *  /home/senecawolf/miniconda3
pytorch                  /home/senecawolf/miniconda3/envs/pytorch

senecawolf@tensorbook:~/Desktop/test-project$ conda activate pytorch
(pytorch) senecawolf@tensorbook:~/Desktop/test-project$ which python
/home/senecawolf/miniconda3/envs/pytorch/bin/python
(pytorch) senecawolf@tensorbook:~/Desktop/test-project$ python
Python 3.9.12 (main, Jun  1 2022, 11:38:51) 
[GCC 7.5.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> 
>>> import torch
>>> print ('\nPytorch versions:', torch.__version__)

Pytorch versions: 1.12.0
>>> torch.version
<module 'torch.version' from '/home/senecawolf/miniconda3/envs/pytorch/lib/python3.9/site-packages/torch/version.py'>
>>> print (torch.version)
<module 'torch.version' from '/home/senecawolf/miniconda3/envs/pytorch/lib/python3.9/site-packages/torch/version.py'>
>>> torch._C._cuda_getCompiledVersion()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: module 'torch._C' has no attribute '_cuda_getCompiledVersion'
>>> print(”\ntorch",torch.file)
  File "<stdin>", line 1
    print(”\ntorch",torch.file)
          ^
SyntaxError: invalid character '”' (U+201D)
>>> print (torch.file)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: module 'torch' has no attribute 'file'
>>> torch.__file__
'/home/senecawolf/miniconda3/envs/pytorch/lib/python3.9/site-packages/torch/__init__.py'
>>> torch.cuda.is_available()
False
>>> device = torch.device(‘cuda’ if torch.cuda.is_available() else ‘cpu’)
  File "<stdin>", line 1
    device = torch.device(‘cuda’ if torch.cuda.is_available() else ‘cpu’)
                          ^
SyntaxError: invalid character '‘' (U+2018)
>>> torch.device('cuda' if torch.cuda.is_available() else 'cpu')
device(type='cpu')
>>> torch.cuda.get_device_name(device)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'device' is not defined
>>> device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
>>> torch.cuda.get_device_name(device)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/senecawolf/miniconda3/envs/pytorch/lib/python3.9/site-packages/torch/cuda/__init__.py", line 329, in get_device_name
    return get_device_properties(device).name
  File "/home/senecawolf/miniconda3/envs/pytorch/lib/python3.9/site-packages/torch/cuda/__init__.py", line 359, in get_device_properties
    _lazy_init()  # will define _get_device_properties
  File "/home/senecawolf/miniconda3/envs/pytorch/lib/python3.9/site-packages/torch/cuda/__init__.py", line 211, in _lazy_init
    raise AssertionError("Torch not compiled with CUDA enabled")
AssertionError: Torch not compiled with CUDA enabled
>>> 
[6]+  Stopped                 python
(pytorch) senecawolf@tensorbook:~/Desktop/test-project$ conda deactivate
senecawolf@tensorbook:~/Desktop/test-project$ python3
Python 3.8.10 (default, Jun 22 2022, 20:18:18) 
[GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch 
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'torch'
>>> device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
[7]+  Stopped                 python3
senecawolf@tensorbook:~/Desktop/test-project$ python
Python 2.7.18 (default, Jul  1 2022, 12:27:04) 
[GCC 9.4.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
>>> torch.cuda.get_device_name(device)
u'NVIDIA GeForce RTX 2070 with Max-Q Design'
>>> 
[8]+  Stopped                 python
senecawolf@tensorbook:~/Desktop/test-project$ 

[markd]

/home/senecawolf/miniconda3/envs/pytorch/lib/python3.9/site-packages/torch/init.py

Okay I see… You have python2 default also even after the upgrade.

On Ubuntu 20.04 the default should be a python3 version.
I cut and paste from here and I had to fix quotes. I have torch 1.11.0 and it does work, so I will have to try 1.12 or see if you have a non CUDA compiled version in Anaconda:

$ python -c ‘import torch ; print("\nPytorch version: ",torch.version) ; print(torch._C._cuda_getCompiledVersion(), “cuda compiled version”)’

Pytorch version: 1.11.0
11060 cuda compiled version

So I would reinstall Lambda stack (I think that was in another thread I just posted that to).
You can email also, since I cannot attach a text file here. (Also it may be easier to do a google meets to work through this).

And we can get python3 set default.

That will show you which python versions are default, setup by links.
On Ubuntu it was using the /etc/alternatives to setup which was installed and set to default.
So we should first check what you have default with:
$ ls -l /usr/bin/python*

But a reinstall of Lambda Stack I would recommend.

[seneca_wolf]

I’m definitely down to do a google meet. I can pay you through paypal for your time.

senecawolf@tensorbook:~$ ls -l /usr/bin/python*
lrwxrwxrwx 1 root root       7 Apr 15  2020 /usr/bin/python -> python2
lrwxrwxrwx 1 root root       9 Mar 13  2020 /usr/bin/python2 -> python2.7
-rwxr-xr-x 1 root root 3662032 Jul  1 07:27 /usr/bin/python2.7
lrwxrwxrwx 1 root root      33 Jul  1 07:27 /usr/bin/python2.7-config -> x86_64-linux-gnu-python2.7-config
lrwxrwxrwx 1 root root      16 Mar 13  2020 /usr/bin/python2-config -> python2.7-config
-rwxr-xr-x 1 root root     385 Apr 11  2018 /usr/bin/python2-futurize
-rwxr-xr-x 1 root root     389 Apr 11  2018 /usr/bin/python2-pasteurize
lrwxrwxrwx 1 root root       9 Mar 13  2020 /usr/bin/python3 -> python3.8
-rwxr-xr-x 1 root root 5502744 Jun 22 15:18 /usr/bin/python3.8
lrwxrwxrwx 1 root root      33 Jun 22 15:18 /usr/bin/python3.8-config -> x86_64-linux-gnu-python3.8-config
lrwxrwxrwx 1 root root      16 Mar 13  2020 /usr/bin/python3-config -> python3.8-config
-rwxr-xr-x 1 root root     384 Mar 27  2020 /usr/bin/python3-futurize
-rwxr-xr-x 1 root root     388 Mar 27  2020 /usr/bin/python3-pasteurize
lrwxrwxrwx 1 root root      14 Apr 15  2020 /usr/bin/python-config -> python2-config
senecawolf@tensorbook:~$ 

[markd]

I would not take payment. I would feel that is a conflict of interest.
As I work at Lambda, and eventually we may have more consulting.

My first priority is just to get you up and running. Since it is not under
a support contract lets do it ‘after hours’.

I am doing a quick install of 18/04 and a upgrade to 20.04, and confirming.
On 18.04:
I confirmed python on 18.04 is pointing to python->python2->python2.7
On 20.04:
And it still points to python2.7

There is a package to make python3 the default:
sudo apt install python-is-python3

And just confirming I think in your case, the lambda stack would need to be reinstalled.
I cannot repeat what you are seeing yet.

Doing the reinstall of Lambda stack will setup python3 as default and make sure everything is up to date.

It is what I normally do, but there other methods that may work.
And once you have it updated, it should be easy to keep up to date with just:

sudo apt-get update && sudo apt-get dist-upgrade

[seneca_wolf]

Okay thank you so much for everything Mark. I’m definitely interesting in doing a video teleconference b/c you might be right it might take less time.

I did the above to make the default python python3 and it is working:

(venv) senecawolf@tensorbook:~/Desktop/test-project$ deactivate 
senecawolf@tensorbook:~/Desktop/test-project$ sudo apt install python-is-python3
[sudo] password for senecawolf: 
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following packages were automatically installed and are no longer required:
  libcublas10.0 libcudart10.0 libcufft10.0 libcurand10.0 libcusparse10.0 libgfortran4 libmagma-dev libmagma2 libnccl2 libnvrtc10.0 libprotoc19 libpython2-dev libpython2.7-dev libsleef3 protobuf-compiler
  python-backports.functools-lru-cache python-bs4 python-cffi python-cffi-backend python-chardet python-configparser python-dateutil python-decorator python-html5lib python-lxml python-mako python-markupsafe
  python-nose python-numpy python-olefile python-pil python-ply python-pycparser python-six python-soupsieve python-typing python-tz python-webencodings python2-dev python2.7-dev
Use 'sudo apt autoremove' to remove them.
The following packages will be REMOVED:
  python-appdirs python-dev-is-python2 python-future python-is-python2 python-pandas python-pandas-lib python-protobuf python-pycuda python-pynvml python-pytools python-scipy python-skcuda python-theano
  python-torch-cuda python-torchvision-cuda
The following NEW packages will be installed:
  python-is-python3
0 upgraded, 1 newly installed, 15 to remove and 0 not upgraded.
Need to get 2,364 B of archives.
After this operation, 1,330 MB disk space will be freed.
Do you want to continue? [Y/n] y
Get:1 http://us.archive.ubuntu.com/ubuntu focal/main amd64 python-is-python3 all 3.8.2-4 [2,364 B]
Fetched 2,364 B in 0s (11.7 kB/s)            
(Reading database ... 364774 files and directories currently installed.)
Removing python-skcuda (0.5.3-0lambda2) ...
Removing python-pycuda (2019.1.2+dfsg-0lambda1) ...
Removing python-pytools (2017.6-1) ...
Removing python-appdirs (1.4.3-1) ...
Removing python-dev-is-python2 (2.7.17-4) ...
Removing python-torchvision-cuda (0.5.0-0lambda1) ...
Removing python-torch-cuda (1.4.0+ds-0lambda2) ...
Removing python-future (0.15.2-4ubuntu2) ...
Removing python-theano (1.0.4+dfsg-0lambda1) ...
Removing python-scipy (0.19.1-2ubuntu1) ...
Removing python-protobuf (3.8.0-0lambda1) ...
Removing python-pynvml (7.352.0-0lambda1) ...
Removing python-pandas (0.22.0-4ubuntu1) ...
Removing python-pandas-lib (0.22.0-4ubuntu1) ...
Removing python-is-python2 (2.7.17-4) ...
Selecting previously unselected package python-is-python3.
(Reading database ... 358956 files and directories currently installed.)
Preparing to unpack .../python-is-python3_3.8.2-4_all.deb ...
Unpacking python-is-python3 (3.8.2-4) ...
Setting up python-is-python3 (3.8.2-4) ...
senecawolf@tensorbook:~/Desktop/test-project$ python
Python 3.8.10 (default, Jun 22 2022, 20:18:18) 
[GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> 
[1]+  Stopped                 python
senecawolf@tensorbook:~/Desktop/test-project$ 

Next, I’m doing:

sudo rm -f /etc/apt/sources.list.d/{graphics,nvidia,cuda}*; COLUMNS=200 dpkg -l | awk \ '/cuda|lib(accinj64|cu(blas|dart|dnn|fft|inj|pti|rand|solver|sparse)|magma|nccl|npp|nv[^p])|nv(idia|ml)|tensor(flow|board)|torch/{ print $2 }' | sudo xargs -or apt -y remove --purge

Which is copy/pasted from: How to upgrade to Ubuntu 20.04 while keeping lambda stack? (I changed the type of quotes used there after getting some errors at the command line).

That looks like it ran correctly (guessing), as seen below:

senecawolf@tensorbook:~/Desktop/test-project$ sudo rm -f /etc/apt/sources.list.d/{graphics,nvidia,cuda}*; COLUMNS=200 dpkg -l | awk \ '/cuda|lib(accinj64|cu(blas|dart|dnn|fft|inj|pti|rand|solver|sparse)|magma|nccl|npp|nv[^p])|nv(idia|ml)|tensor(flow|board)|torch/{ print $2 }' | sudo xargs -or apt -y remove --purge
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following packages were automatically installed and are no longer required:
  libatomic1:i386 libbsd0:i386 libdrm-amdgpu1:i386 libdrm-intel1:i386 libdrm-nouveau2:i386 libdrm-radeon1:i386 libdrm2:i386 libedit2:i386 libelf1:i386 libexpat1:i386 libffi7:i386 libgfortran4 libgl1:i386
  libgl1-mesa-dri:i386 libglapi-mesa:i386 libglvnd0:i386 libglx-mesa0:i386 libglx0:i386 libllvm12:i386 libpciaccess0:i386 libprotoc19 libpython2-dev libpython2.7-dev libsensors5:i386 libsleef3 libstdc++6:i386
  libthrust-dev libvdpau-dev libvdpau1 libvulkan1:i386 libwayland-client0:i386 libx11-6:i386 libx11-xcb1:i386 libxau6:i386 libxcb-dri2-0:i386 libxcb-dri3-0:i386 libxcb-glx0:i386 libxcb-present0:i386
  libxcb-randr0:i386 libxcb-shm0:i386 libxcb-sync1:i386 libxcb-xfixes0:i386 libxcb1:i386 libxdmcp6:i386 libxext6:i386 libxfixes3:i386 libxshmfence1:i386 libxxf86vm1:i386 mesa-vdpau-drivers
  mesa-vulkan-drivers:i386 node-html5shiv nsight-compute nsight-systems protobuf-compiler python-backports.functools-lru-cache python-bs4 python-cffi python-cffi-backend python-chardet python-configparser
  python-dateutil python-decorator python-html5lib python-lxml python-mako python-markupsafe python-nose python-numpy python-olefile python-pil python-ply python-pycparser python-six python-soupsieve
  python-typing python-tz python-webencodings python2-dev python2.7-dev vdpau-driver-all
Use 'sudo apt autoremove' to remove them.
The following packages will be REMOVED:
  libaccinj64-10.1* libcublas10* libcublas10.0* libcublaslt10* libcudart10.0* libcudart10.1* libcudnn-dev* libcudnn7* libcufft10* libcufft10.0* libcufftw10* libcuinj64-10.1* libcupti-dev* libcupti-doc*
  libcupti10.1* libcurand10* libcurand10.0* libcusolver10* libcusolvermg10* libcusparse10* libcusparse10.0* libmagma-dev* libmagma2* libnccl2* libnppc10* libnppial10* libnppicc10* libnppicom10* libnppidei10*
  libnppif10* libnppig10* libnppim10* libnppist10* libnppisu10* libnppitc10* libnpps10* libnvblas10* libnvgraph10* libnvidia-cfg1-510* libnvidia-common-510* libnvidia-compute-418* libnvidia-compute-418:i386*
  libnvidia-compute-510* libnvidia-compute-510:i386* libnvidia-decode-510* libnvidia-decode-510:i386* libnvidia-encode-510* libnvidia-encode-510:i386* libnvidia-extra-510* libnvidia-fbc1-510*
  libnvidia-fbc1-510:i386* libnvidia-gl-510* libnvidia-gl-510:i386* libnvidia-ml-dev* libnvjpeg10* libnvrtc10.0* libnvrtc10.1* libnvtoolsext1* libnvvm3* nvidia-compute-utils-510* nvidia-cuda-dev*
  nvidia-cuda-doc* nvidia-cuda-gdb* nvidia-cuda-toolkit* nvidia-dkms-510* nvidia-driver-410* nvidia-driver-418* nvidia-driver-430* nvidia-driver-440* nvidia-driver-450* nvidia-driver-455* nvidia-driver-460*
  nvidia-driver-470* nvidia-driver-510* nvidia-kernel-common-510* nvidia-kernel-source-510* nvidia-prime* nvidia-profiler* nvidia-settings* nvidia-utils-510* nvidia-visual-profiler* python-torch-cuda*
  python3-pynvml* python3-torch-cuda* screen-resolution-extra* xserver-xorg-video-nvidia-510*
0 upgraded, 0 newly installed, 86 to remove and 0 not upgraded.
After this operation, 5,102 MB disk space will be freed.
(Reading database ... 358960 files and directories currently installed.)
Removing libcudnn-dev:amd64 (7.6.5-0lambda1) ...
Removing nvidia-cuda-toolkit (10.1.243-3) ...
Removing nvidia-cuda-dev (10.1.243-3) ...
Removing nvidia-visual-profiler (10.1.243-3) ...
Removing nvidia-profiler (10.1.243-3) ...
Removing libaccinj64-10.1:amd64 (10.1.243-3) ...
Removing libnvblas10:amd64 (10.1.243-3) ...
Removing libcublas10:amd64 (10.1.243-3) ...
Removing libmagma-dev:amd64 (2.5.2+ds-0lambda1) ...
Removing libmagma2:amd64 (2.5.2+ds-0lambda1) ...
Removing libcublas10.0:amd64 (10.0.130-0lambda3) ...
Removing libcublaslt10:amd64 (10.1.243-3) ...
Removing libnccl2:amd64 (2.5.7-0lambda1) ...
Removing libcudart10.0:amd64 (10.0.130-0lambda3) ...
Removing libcudart10.1:amd64 (10.1.243-3) ...
Removing libcudnn7:amd64 (7.6.5-0lambda1) ...
Removing libcufftw10:amd64 (10.1.243-3) ...
Removing libcufft10:amd64 (10.1.243-3) ...
Removing libcufft10.0:amd64 (10.0.130-0lambda3) ...
Removing libcuinj64-10.1:amd64 (10.1.243-3) ...
Removing libcupti-dev:amd64 (10.1.243-3) ...
Removing libcupti-doc (10.1.243-3) ...
Removing libcupti10.1:amd64 (10.1.243-3) ...
Removing libnvgraph10:amd64 (10.1.243-3) ...
Removing libcurand10:amd64 (10.1.243-3) ...
Removing libcurand10.0:amd64 (10.0.130-0lambda3) ...
Removing libcusolver10:amd64 (10.1.243-3) ...
Removing libcusolvermg10:amd64 (10.1.243-3) ...
Removing libcusparse10:amd64 (10.1.243-3) ...
Removing libcusparse10.0:amd64 (10.0.130-0lambda3) ...
Removing libnppial10:amd64 (10.1.243-3) ...
Removing libnpps10:amd64 (10.1.243-3) ...
Removing libnppicc10:amd64 (10.1.243-3) ...
Removing libnppicom10:amd64 (10.1.243-3) ...
Removing libnppidei10:amd64 (10.1.243-3) ...
Removing libnppif10:amd64 (10.1.243-3) ...
Removing libnppig10:amd64 (10.1.243-3) ...
Removing libnppim10:amd64 (10.1.243-3) ...
Removing libnppist10:amd64 (10.1.243-3) ...
Removing libnppisu10:amd64 (10.1.243-3) ...
Removing libnppitc10:amd64 (10.1.243-3) ...
Removing nvidia-driver-410 (418.113-0lambda0~18.04.3) ...
Removing nvidia-driver-418 (430.64-0lambda0~18.04.2) ...
Removing nvidia-driver-430 (440.100-0ubuntu0.20.04.1) ...
Removing nvidia-driver-440 (450.119.03-0ubuntu0.20.04.1) ...
Removing nvidia-driver-450 (460.91.03-0ubuntu0.20.04.1) ...
Removing nvidia-driver-455 (460.91.03-0ubuntu0.20.04.1) ...
Removing nvidia-driver-460 (470.129.06-0ubuntu0.20.04.1) ...
Removing nvidia-driver-470 (510.73.05-0lambda0~18.04.1) ...
Removing nvidia-driver-510 (510.73.05-0ubuntu0.20.04.1) ...
Removing xserver-xorg-video-nvidia-510 (510.73.05-0ubuntu0.20.04.1) ...
Removing libnvidia-cfg1-510:amd64 (510.73.05-0ubuntu0.20.04.1) ...
Removing libnvidia-gl-510:amd64 (510.73.05-0ubuntu0.20.04.1) ...
Removing libnvidia-gl-510:i386 (510.73.05-0ubuntu0.20.04.1) ...
Removing libnvidia-common-510 (510.73.05-0ubuntu0.20.04.1) ...
Removing libnvidia-encode-510:i386 (510.73.05-0ubuntu0.20.04.1) ...
Removing libnvidia-decode-510:i386 (510.73.05-0ubuntu0.20.04.1) ...
Removing libnvidia-compute-510:i386 (510.73.05-0ubuntu0.20.04.1) ...
Removing libnvidia-encode-510:amd64 (510.73.05-0ubuntu0.20.04.1) ...
Removing libnvidia-decode-510:amd64 (510.73.05-0ubuntu0.20.04.1) ...
Removing libnvidia-ml-dev (10.1.243-3) ...
Removing libnvrtc10.0:amd64 (10.0.130-0lambda3) ...
Removing nvidia-utils-510 (510.73.05-0ubuntu0.20.04.1) ...
Removing python3-pynvml (7.352.0-3) ...
Removing libnvidia-extra-510:amd64 (510.73.05-0ubuntu0.20.04.1) ...
Removing libnvidia-fbc1-510:amd64 (510.73.05-0ubuntu0.20.04.1) ...
Removing libnvidia-fbc1-510:i386 (510.73.05-0ubuntu0.20.04.1) ...
Removing libnvjpeg10:amd64 (10.1.243-3) ...
Removing libnvrtc10.1:amd64 (10.1.243-3) ...
Removing libnvtoolsext1:amd64 (10.1.243-3) ...
Removing libnvvm3:amd64 (10.1.243-3) ...
Removing nvidia-compute-utils-510 (510.73.05-0ubuntu0.20.04.1) ...
Removing nvidia-cuda-doc (10.1.243-3) ...
Removing nvidia-cuda-gdb (10.1.243-3) ...
Removing nvidia-dkms-510 (510.73.05-0ubuntu0.20.04.1) ...
Removing all DKMS Modules
Done.
INFO:Disable nvidia
DEBUG:Parsing /usr/share/ubuntu-drivers-common/quirks/dell_latitude
DEBUG:Parsing /usr/share/ubuntu-drivers-common/quirks/put_your_quirks_here
DEBUG:Parsing /usr/share/ubuntu-drivers-common/quirks/lenovo_thinkpad
update-initramfs: deferring update (trigger activated)
Removing nvidia-kernel-common-510 (510.73.05-0ubuntu0.20.04.1) ...
update-initramfs: deferring update (trigger activated)
Removing nvidia-kernel-source-510 (510.73.05-0ubuntu0.20.04.1) ...
Removing nvidia-prime (0.8.16~0.20.04.2) ...
Removing nvidia-settings (510.73.05-0lambda1) ...
Removing screen-resolution-extra (0.18build1) ...
Removing libnppc10:amd64 (10.1.243-3) ...
Removing libnvidia-compute-510:amd64 (510.73.05-0ubuntu0.20.04.1) ...
Processing triggers for mime-support (3.64ubuntu1) ...
Processing triggers for initramfs-tools (0.136ubuntu6.7) ...
update-initramfs: Generating /boot/initrd.img-5.4.0-122-generic
Processing triggers for gnome-menus (3.36.0-1ubuntu1) ...
Processing triggers for libc-bin (2.31-0ubuntu9.9) ...
Processing triggers for man-db (2.9.1-1) ...
Processing triggers for desktop-file-utils (0.24-1ubuntu3) ...
(Reading database ... 350292 files and directories currently installed.)
Purging configuration files for libcudnn-dev:amd64 (7.6.5-0lambda1) ...
Purging configuration files for nvidia-dkms-510 (510.73.05-0ubuntu0.20.04.1) ...
update-initramfs: deferring update (trigger activated)
Purging configuration files for libcudnn7:amd64 (7.6.5-0lambda1) ...
Purging configuration files for nvidia-prime (0.8.16~0.20.04.2) ...
Purging configuration files for libnvidia-compute-418:amd64 (430.64-0lambda0~18.04.1) ...
Purging configuration files for libnvidia-compute-418:i386 (430.64-0lambda0~18.04.1) ...
Purging configuration files for libnvidia-compute-510:amd64 (510.73.05-0ubuntu0.20.04.1) ...
Purging configuration files for libnvidia-compute-510:i386 (510.73.05-0ubuntu0.20.04.1) ...
Purging configuration files for nvidia-cuda-toolkit (10.1.243-3) ...
Purging configuration files for nvidia-compute-utils-510 (510.73.05-0ubuntu0.20.04.1) ...
Purging configuration files for screen-resolution-extra (0.18build1) ...
Purging configuration files for nvidia-settings (510.73.05-0lambda1) ...
Purging configuration files for python-torch-cuda (1.4.0+ds-0lambda2) ...
Purging configuration files for python3-torch-cuda (1.4.0+ds-0lambda2) ...
Purging configuration files for nvidia-kernel-common-510 (510.73.05-0ubuntu0.20.04.1) ...
update-initramfs: deferring update (trigger activated)
Processing triggers for dbus (1.12.16-2ubuntu2.2) ...
Processing triggers for initramfs-tools (0.136ubuntu6.7) ...
update-initramfs: Generating /boot/initrd.img-5.4.0-122-generic
senecawolf@tensorbook:~/Desktop/test-project$ 

And next I am doing:

LAMBDA_REPO=$(mktemp) && wget -O${LAMBDA_REPO} https://lambdalabs.com/static/misc/lambda-stack-repo.deb && sudo dpkg -i ${LAMBDA_REPO} && rm -f ${LAMBDA_REPO} && sudo apt-get -y update && sudo apt-get -y install lambda-stack-cuda

Which I copy/pasted from: How to upgrade to Ubuntu 20.04 while keeping lambda stack?

And then I did:

sudo apt-get update && sudo apt-get dist-upgrade

Which I got from earlier on this same page.

Finally, it looks like some of the above are working properly (I wonder why copy/pasting from this forum changes quotes to not work when you copy paste them at the terminal?):

senecawolf@tensorbook:~$ python
Python 3.8.10 (default, Jun 22 2022, 20:18:18) 
[GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> torch.__version__
'1.11.0'
>>> torch.cuda.is_available()
True
>>> torch.__file__
'/usr/lib/python3/dist-packages/torch/__init__.py'
>>> torch._C._cuda_getCompiledVersion()
11060
>>> device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
>>> torch.cuda.get_device_name(device)
'NVIDIA GeForce RTX 2070 with Max-Q Design'
>>>