Fresh lambda-stack fails python help("modules") CUBLAS error

rneely · July 8, 2021, 9:48pm

Hi,
I installed the latest lambda stack on a fresh Ubuntu 20.04 per Lambda Stack: an AI software stack that's always up-to-date. I then opened a bash terminal and ran the following

$ python -c “help(‘modules’)”

which produced a CUBLAS errror. I am trying to see a list of python modules that are installed from within python.

Please wait a moment while I gather a list of all available modules…

/usr/lib/python3/dist-packages/IPython/kernel/init.py:12: ShimWarning: The IPython.kernel package has been deprecated since IPython 4.0.You should import from ipykernel or jupyter_client instead.
warn(“The IPython.kernel package has been deprecated since IPython 4.0.”
Unable to init server: Could not connect: Connection refused
Unable to init server: Could not connect: Connection refused
/usr/lib/python3/dist-packages/UpdateManager/backend/init.py:11: PyGIWarning: Gtk was imported without specifying a version first. Use gi.require_version(‘Gtk’, ‘3.0’) before import to ensure that the right version gets loaded.
from gi.repository import GLib, Gtk, Snapd
No protocol specified
Using TensorFlow backend.
2021-07-08 21:38:46.071513: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudart.so.11.0
/usr/lib/python3.8/pkgutil.py:92: MatplotlibDeprecationWarning:
The mpl_toolkits.axes_grid module was deprecated in Matplotlib 2.1 and will be removed two minor releases later. Use mpl_toolkits.axes_grid1 and mpl_toolkits.axisartist, which provide the same functionality instead.
import(info.name)

(-c:12342): Gdk-CRITICAL **: 21:38:48.418: gdk_cursor_new_for_display: assertion ‘GDK_IS_DISPLAY (display)’ failed
/usr/lib/python3.8/pkgutil.py:92: UserWarning: Viewer requires Qt
import(info.name)
/usr/lib/python3/dist-packages/skcuda/cublas.py:284: UserWarning: creating CUBLAS context to get version number
warnings.warn(‘creating CUBLAS context to get version number’)
[libprotobuf ERROR google/protobuf/descriptor_database.cc:318] Invalid file descriptor data passed to EncodedDescriptorDatabase::Add().
[libprotobuf FATAL google/protobuf/descriptor.cc:1358] CHECK failed: GeneratedDatabase()->Add(encoded_file_descriptor, size):
terminate called after throwing an instance of ‘google::protobuf::FatalException’
what(): CHECK failed: GeneratedDatabase()->Add(encoded_file_descriptor, size):
*** Process received signal ***
Signal: Aborted (6)
Signal code: (-6)
[ 0] /lib/x86_64-linux-gnu/libc.so.6(+0x46210)[0x7f871448f210]
[ 1] /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xcb)[0x7f871448f18b]
[ 2] /lib/x86_64-linux-gnu/libc.so.6(abort+0x12b)[0x7f871446e859]
[ 3] /lib/x86_64-linux-gnu/libstdc++.so.6(+0x9e911)[0x7f86c1708911]
[ 4] /lib/x86_64-linux-gnu/libstdc++.so.6(+0xaa38c)[0x7f86c171438c]
[ 5] /lib/x86_64-linux-gnu/libstdc++.so.6(+0xaa3f7)[0x7f86c17143f7]
[ 6] /lib/x86_64-linux-gnu/libstdc++.so.6(+0xaa6a9)[0x7f86c17146a9]
[ 7] /lib/x86_64-linux-gnu/libprotobuf.so.17(+0xbff7c)[0x7f84b3922f7c]
[ 8] /lib/x86_64-linux-gnu/libprotobuf.so.17(_ZN6google8protobuf14DescriptorPool24InternalAddGeneratedFileEPKvi+0x8a)[0x7f84b397df1a]
[ 9] /lib/x86_64-linux-gnu/libprotobuf.so.17(_ZN40protobuf_google_2fprotobuf_2fany_2eproto18AddDescriptorsImplEv+0x1e)[0x7f84b3974dae]
[10] /lib/x86_64-linux-gnu/libpthread.so.0(+0x1247f)[0x7f871443847f]
[11] /lib/x86_64-linux-gnu/libprotobuf.so.17(+0x112b69)[0x7f84b3975b69]
[12] /lib64/ld-linux-x86-64.so.2(+0x11b8a)[0x7f871466bb8a]
[13] /lib64/ld-linux-x86-64.so.2(+0x11c91)[0x7f871466bc91]
[14] /lib/x86_64-linux-gnu/libc.so.6(_dl_catch_exception+0xe5)[0x7f87145ac915]
[15] /lib64/ld-linux-x86-64.so.2(+0x160bf)[0x7f87146700bf]
[16] /lib/x86_64-linux-gnu/libc.so.6(_dl_catch_exception+0x88)[0x7f87145ac8b8]
[17] /lib64/ld-linux-x86-64.so.2(+0x155fa)[0x7f871466f5fa]
[18] /lib/x86_64-linux-gnu/libdl.so.2(+0x134c)[0x7f871442134c]
[19] /lib/x86_64-linux-gnu/libc.so.6(_dl_catch_exception+0x88)[0x7f87145ac8b8]
[20] /lib/x86_64-linux-gnu/libc.so.6(_dl_catch_error+0x33)[0x7f87145ac983]
[21] /lib/x86_64-linux-gnu/libdl.so.2(+0x1b59)[0x7f8714421b59]
[22] /lib/x86_64-linux-gnu/libdl.so.2(dlopen+0x4a)[0x7f87144213da]
[23] python(_PyImport_FindSharedFuncptr+0x6f)[0x6737cf]
[24] python(_PyImport_LoadDynamicModuleWithSpec+0x15b)[0x6840db]
[25] python[0x684c71]
[26] python[0x5c3cb0]
[27] python(PyVectorcall_Call+0x58)[0x5f2168]
[28] python(_PyEval_EvalFrameDefault+0x6552)[0x570902]
[29] python(_PyEval_EvalCodeWithName+0x26a)[0x568d9a]
*** End of error message ***
Aborted

Running the following from the bash terminal

python3 -c “import torch;print(torch.version);print(torch.cuda.is_available());print(torch.cuda.get_device_name(torch.cuda.current_device()));”

shows

1.9.0
True
Tesla T4

I tried upgrading the lambda-stack with

sudo apt-get update && sudo apt-get dist-upgrade

and it seemed to work and printed no errors.

However, I keep getting the same failure on help(:modules").

How can I get a list of python modules from within python with lambda-stack?

Is this the right place to report this? Is there a git repo I should attempt to report it on?

markd · July 30, 2021, 3:56pm

Sorry for the delay.

I confirmed the help(modules) works with the clean lambda stack.
python -c "help('modules')"
It seems you have a different iPython that is deprecated. And then there are Google protocol buffer differences.
- So it looks like you have a mix of old software.
To remove/clean up lambda stack, but also other old python packages, you can do the following:

The following will remove all NVidia software, deep learning libraries, and then
reinstall Lambda Stack.

sudo rm -f /etc/apt/sources.list.d/{graphics,nvidia,cuda}*; \
	COLUMNS=200 dpkg -l |
	awk '/cuda|lib(accinj64|cu(blas|dart|dnn|fft|inj|pti|rand|solver|sparse)|magma|nccl|npp|nv[^p])|nv(idia|ml)|tensor(flow|board)|torch/ { print $2 }' |
	sudo xargs -or apt -y remove --purge
    # Then it reinstalls Lambda Stack
    LAMBDA_REPO=$(mktemp) && \
	wget -O${LAMBDA_REPO} https://lambdalabs.com/static/misc/lambda-stack-repo.deb && \
	sudo dpkg -i ${LAMBDA_REPO} && rm -f ${LAMBDA_REPO} && \
	sudo apt-get -y update && sudo apt-get -y install lambda-stack-cuda

If that does not help, I can help you work through this.

Mark