That is a good point.
I should have mentioned it is located in /usr/lib/python3/dist-packages/tensorflow if you have installed Lambda Stack for libcudnn_ops_train.so.8
If you do not have lambda stack you would need to install cudnn from NVIDIA, where you need to register to get to the cudnn download page.
Cudnn: CUDA Deep Neural Network (cuDNN) | NVIDIA Developer
And you need to make sure wherever you install it the LD_LIBRARY_PATH can be seen. Also so that the version is compatible with the driver and CUDA you have installed.
If you are running inside of anaconda. Anaconda does not setup the library paths for packages.
You should be able to find the path, the likely locations would be:
- /usr
- /usr/local - that is only suppose to be used for local sites software
- /opt/<vendor>/<product>/<version> - standard for Vendor software
- ~/Anaconda3 or I use: find ${CONDA_PREFIX} -name ‘libcudnn.so*’
LD_LIBRARY_PATH=${CONDA_PREFIX}/lib:${LD_LIBRARY_PATH} jupyter notebook
Let me know if that helps.
Or if you run directly from python (with the nvidia-smi lines commented out)
python LSTM-hell.py | tee -a LSTM-hell.txt
And send the LSTM-hell.txt