Dist-upgrade does not upgrade to Cuda 11.1

[dkkim93]

Hello. First of all, thank you to the Lambda team for providing the Lambda stack package!
I would like to install the Cuda 11.1 on my Ubuntu 18.04 with RTX 3090 to run a PyTorch program with GPU.
I followed the instruction and first installed by:

LAMBDA_REPO=$(mktemp) && \
wget -O${LAMBDA_REPO} https://lambdalabs.com/static/misc/lambda-stack-repo.deb && \
sudo dpkg -i ${LAMBDA_REPO} && rm -f ${LAMBDA_REPO} && \
sudo apt-get update && sudo apt-get install -y lambda-stack-cuda
sudo reboot

and then
sudo apt-get update && sudo apt-get dist-upgrade.

However, unfortunately, the installed Cuda version is still 10.0, and I would like to ask for help.
I appreciate your time.

Additional information:

python3                                                                  6s
Python 3.6.9 (default, Oct  8 2020, 12:12:24) 
[GCC 8.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> torch.__version__
'1.4.0'

nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2018 NVIDIA Corporation
Built on Sat_Aug_25_21:08:01_CDT_2018
Cuda compilation tools, release 10.0, V10.0.130

nvidia-smi
Thu Dec 10 09:33:53 2020       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 455.45.01    Driver Version: 455.45.01    CUDA Version: 11.1     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  GeForce RTX 3090    On   | 00000000:01:00.0  On |                  N/A |
| 32%   34C    P8    35W / 350W |    360MiB / 24265MiB |     31%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A      1260      G   /usr/lib/xorg/Xorg                164MiB |
|    0   N/A  N/A      1405      G   /usr/bin/gnome-shell               41MiB |
|    0   N/A  N/A      1902      G   ...AAAAAAAA== --shared-files      150MiB |
+-----------------------------------------------------------------------------+

[jeremy]

The latest version of the CUDA toolkit we support on Ubuntu 18.04 is version 10.0.130.

If you want to use version 11.1, you’ll need to upgrade to Ubuntu 20.04.
You can do that like so:

sudo sed -i '/^AutomaticLoginEnable/ s/true/false/' /etc/gdm3/custom.conf &&\
sudo apt -y update && sudo apt -y dist-upgrade && sudo do-release-upgrade -d -f DistUpgradeViewNonInteractive

Then, once the release upgrade completes, and the machine reboots, run the following to reinstall Lambda Stack:

LAMBDA_REPO=$(mktemp) && \
	wget -O${LAMBDA_REPO} https://lambdalabs.com/static/misc/lambda-stack-repo.deb && \
	sudo dpkg -i ${LAMBDA_REPO} && rm -f ${LAMBDA_REPO} && \
	sudo apt-get update && sudo apt-get install -y lambda-stack-cuda

[Karun]

After I upgraded (with several hoops and tries) to Ubuntu 20.04 Desktop (from 18.04) via
sudo sed -i ‘/^AutomaticLoginEnable/ s/true/false/’ /etc/gdm3/custom.conf &&
sudo apt -y update && sudo apt -y dist-upgrade && sudo do-release-upgrade -d -f DistUpgradeViewNonInteractive

And then rebooted and installed lambda stack (again I had to try a few times) via

LAMBDA_REPO=$(mktemp) &&
wget -O${LAMBDA_REPO} https://lambdalabs.com/static/misc/lambda-stack-repo.deb &&
sudo dpkg -i ${LAMBDA_REPO} && rm -f ${LAMBDA_REPO} &&
sudo apt-get update && sudo apt-get install -y lambda-stack-cuda

I have two issues:

1. I am unable to run the command ‘nvidia-smi’ as I am getting communication error as shown below.
   “NVIDIA-SMI has failed because it couldn’t communicate with the NVIDIA driver. Make sure that the latest NVIDIA driver is installed and running.”

So when I want to install
$sudo apt install nvidia-prime
But it is failing with problem in not able to install lambda-desktop : Depends: python-is-python3 but it is not going to be installed
You might want to run ‘apt --fix-broken install’ to correct these.
The following packages have unmet dependencies:
lambda-desktop : Depends: python-is-python3 but it is not going to be installed
E: Unmet dependencies. Try ‘apt --fix-broken install’ with no packages (or specify a solution).

So I tried installing
The following NEW packages will be installed:
python-is-python3
and that resulted in
Errors were encountered while processing:
** /var/cache/apt/archives/python-is-python3_3.8.2-4_all.deb**
E: Sub-process /usr/bin/dpkg returned an error code (1)

Any idea how to overcome this? I tried several solutions and still not able to proceed.
Thanks

[Karun]

Just a hunch, but I’m thinking there still might be some python*-*dev packages installed that are blocking this.
Can you send me the output of
dpkg -l | grep python
$ dpkg -l | grep python | grep minimal
ii libpython2.7-minimal:amd64 2.7.18-1~20.04.1 amd64 Minimal subset of the Python language (version 2.7)
ii libpython3.6-minimal:amd64 3.6.9-1~18.04ubuntu1.7 amd64 Minimal subset of the Python language (version 3.6)
ii libpython3.7-minimal:amd64 3.7.5-2ubuntu1~18.04.2 amd64 Minimal subset of the Python language (version 3.7)
ii libpython3.8-minimal:amd64 3.8.10-0ubuntu1~20.04.4 amd64 Minimal subset of the Python language (version 3.8)
ii python-minimal 2.7.15~rc1-1 amd64 minimal subset of the Python language (default version)
ii python2.7-minimal 2.7.18-1~20.04.1 amd64 Minimal subset of the Python language (version 2.7)
ii python3-minimal 3.8.2-0ubuntu2 amd64 minimal subset of the Python language (default python3 version)
ii python3.6-minimal 3.6.9-1~18.04ubuntu1.7 amd64 Minimal subset of the Python language (version 3.6)
ii python3.7-minimal 3.7.5-2ubuntu1~18.04.2 amd64 Minimal subset of the Python language (version 3.7)
ii python3.8-minimal 3.8.10-0ubuntu1~20.04.4 amd64 Minimal subset of the Python language (version 3.8)

Hi Matt,
I removed the offending python-minimal 2.7.15~rc1-1 amd64 minimal subset of the Python language (default version
in /var/lib/dpkg/status as per this https://stackoverflow.com/questions/58462509/unable-to-install-python-dev
and did sudo apt-get install -f python-is-python3 .
Now I cleaned config via sudo dpkg --configure -a and sudo apt --fix-broken install and all went well.
And now it seems straighntened. Is it safe to follow the clean install that you recommended again?
Now nividia-smi is working as expected.
Thanks