Lambda Stack - CUDNN 8 Upgrade Query

subhrajeitbhowmick 2020-11-06 07:34:39 UTC #1

Hi,

I use a Tensorbook and need to leverage on Tensorflow GPU support for CUDA 11. Though the latest Lambda Stack upgrade switched my previous CUDA 10.2 to 11.1, the CUDNN version still remains 7.6.

Do we know of a timeline by when we can expect Lambda Stack to upgrade its CUDNN 7.6 to CUDNN 8.x?

Alternatively, is there a suggestion how to upgrade it manually without breaking the Lambda Stack - ensuring no compatibility issues in future upgrades of the Lambda Stack?

Many thanks in advance!

Regards.

sabalaba 2020-11-13 17:45:04 UTC #2

Lambda Stack with CuDNN 8 is coming shortly.

In the mean time, you should be able to link to CUDNN with something like modifying your LD_LIBRARY_PATH to include a path to the libcudnn.so.8 files.

subhrajeitbhowmick 2020-11-14 21:54:05 UTC #3

Many thanks @sabalaba for your kind response!

I downloaded cuDNN 8 for Ubuntu 20.04 and added the LD_LIBRARY_PATH as per your suggestion.

However, while testing for Tensorflow’s GPU association, I received an error stating, “Could not load dynamic library ‘libcusolver.so.10’; dlerror: libcusolver.so.10: cannot open shared object file: No such file or directory”.

This might be resolved by completely removing all installed CUDA files and a fresh install of CUDA, but I fear, this would lead to an incongruity in the existing Lambda Stack.

Note: I even tried with Tensorflow 2.4.0-rc1 whose pip packages are now built with CUDA11 and cuDNN 8.0.2.

Would you kindly be able to suggest a solution here?

perreiradasilva-m 2020-11-18 11:53:18 UTC #4

Hello to all,

Just to add more details @sabalaba (because I faced the same problem), it seems there is however a problem with the current lambdastack version. On a fresh install, any pytorch call to code that uses cudnn throws an error :
“Could not load library libcudnn_cnn_train.so.8. Error: libcudnn_ops_train.so.8: cannot open shared object file: No such file or directory
Please make sure libcudnn_cnn_train.so.8 is in your library path!”

The only way to get things working is to set LD_LIBRARY_PATH accordingly:
export LD_LIBRARY_PATH=/usr/lib/python3/dist-packages/torch/lib/

But is seem not to be that lambda stack way.

I understand that cudnn 8 is not yet supported but in that case why does a fresh install try to use cudnn 8 by default ?