Ubuntu 22.04 support timeline

[tjoslin]

Is there a timeline on when the lambda stack will be fully supported for Ubuntu 22.04 as it is now the current LTS version?

[markd]

Yes Ubuntu 22.04 LTS was just released in April 2022. It is in testing currently.

I do not have a official date. This is something we have been perusing since it was released.
It is normally best not to move on to the new OS until it has had a few months of burn-in time and packages, libraries, software stacks have been ported, tested and released.

This is much more solid than on the non-LTS branch, however, this is where the more production machines and applications are tested.

Once it is released, or I have a official time line I will try to update you here.

Mark

[tjoslin]

I’d much rather have things thoroughly tested, so I fully understand. Thanks for the reply.

[markd]

Excellent news. They feel it is ready.

I am doing a update right now to test, and I will let you know in about 2 hours how it worked out.

I am doing the following from Ubuntu 20.04.4 LTS Desktop.
$ sudo apt-get update && sudo apt-get dist-upgrade
$ sudo reboot # If there are changes since my last reboot

$ sudo do-release-upgrade -d --allow-third-party

Then I will do further testing:
* pytorch
* Tensorflow
* docker/docker-compose (docker.io version not the old nvidia-docker2)
* And run some codes directly on the machine

And I will send a update here.

[markd]

I installed, and basic things all worked the same for:
nvidia-smi
basic python codes (tensorflow, pytorch)

Docker I could not get started without a update.

• Was getting the error:
  docker: Error response from daemon: OCI runtime create failed: container_linux.go:370: starting container process caused: process_linux.go:459: container init caused: Running hook #0:: error running hook: exit status 1, stdout: , stderr: nvidia-container-cli: container error: cgroup subsystem devices not found: unknown.

The Fix/Workaround was to add a kernel boot option:
$ sudo sed -iE ‘/^GRUB_CMDLINE.*DEFAULT/ s/"$/ systemd.unified_cgroup_hierarchy=false"/’ /etc/default/grub
$ sudo update-grub
$ sudo reboot

Then docker was happily working.

[tonyr]

I installed, and basic things all worked the same

Thanks a lot for posting this, I’ve just done the same and found the same. I was forced to upgrade, import torchvision started throwing some C++11 incompatabiliies. Luckily the upgrade was very smooth, (bar X11 permission denied messages obscuring the option choices). I’m still left with a dud gridengine, but there again which LTS version of Ubuntu hasn’t needed a gridengine bodge?

[ncowan]

Is it advisable to update my lambdalabs notebook to 22.04LTS? Has it all been thoroughly tested?

[markd]

It has been tested. And I have only run into one issue with docker on it. And Lambda Stack was updated to deal with that. The issue was related to cgroups v2 usage. And if you have a issue I previously had changed the kernel parameters, by just adding ‘systemd.unified_cgroup_hierarchy=false’, that gets added to the ‘GRUB_CMDLINE_LINUX_DEFAULT=’ line in /etc/default/grub, then run: ‘sudo update-grub’

For example:
sudo sed -iE ‘/^GRUB_CMDLINE.*DEFAULT/ s/"$/ systemd.unified_cgroup_hierarchy=false"/’ /etc/default/grub
sudo update-grub

We are shipping desktops with this, but not servers yet with 22.04 LTS.

[votbod]

Update and reboot seemed to work fine but do-release-upgrade… fails with status 1 and only gives me option to terminate or resurrect. resurrect just cycles back to same place. 20.04.6 LTS.

…:~$ sudo do-release-upgrade -d --allow-third-party
Checking for a new Ubuntu release
Get:1 Upgrade tool signature [819 B]
Get:2 Upgrade tool [1,266 kB]
Fetched 1,267 kB in 0s (0 B/s)
authenticate ‘jammy.tar.gz’ against 'jammy.tar.gz.gpg’
extracting ‘jammy.tar.gz’
[screen is terminating]
…:~$

[markd]

I would not use Ubuntu’s --allow-third-party, as that will attempt to install their versions of the nvidia drivers, cuda, etc. Versus the version that pytorch and tensorflow are built with.

There would be more information in the logs about the conflicts.

Also you should do:
$ sudo apt update
$ sudo apt dist-upgrade
Then do the:
$ sudo do-release-upgrade -d

Or you can first remove lambda stack, do the upgrade, then do the install.
$ sudo apt update
$ sudo apt dist-upgrade
… then remove lambda stack
https://docs.lambdalabs.com/linux/remove-reinstall-lambda-stack/

sudo rm -f /etc/apt/sources.list.d/{graphics,nvidia,cuda}* &&
dpkg -l |
awk ‘/cuda|lib(accinj64|cu(blas|dart|dnn|fft|inj|pti|rand|solver|sparse)|magma|nccl|npp|nv[^p])|nv(idia|ml)|tensor(flow|board)|torch/ { print $2 }’ |
sudo xargs -or apt -y remove --purge

Do the upgrade:
sudo do-release-upgrade -d

Then reinstall lambda stack: (https://lambdalabs.com/lambda-stack-deep-learning-software)
wget -nv -O- https://lambdalabs.com/install-lambda-stack.sh | sh -

Then do the reboot.