Fresh lambda-stack fails python help("modules") CUBLAS error

I installed the latest lambda stack on a fresh Ubuntu 20.04 per Lambda Stack: an AI software stack that's always up-to-date. I then opened a bash terminal and ran the following

$ python -c “help(‘modules’)”

which produced a CUBLAS errror. I am trying to see a list of python modules that are installed from within python.

Please wait a moment while I gather a list of all available modules…

/usr/lib/python3/dist-packages/IPython/kernel/ ShimWarning: The IPython.kernel package has been deprecated since IPython 4.0.You should import from ipykernel or jupyter_client instead.
warn(“The IPython.kernel package has been deprecated since IPython 4.0.”
Unable to init server: Could not connect: Connection refused
Unable to init server: Could not connect: Connection refused
/usr/lib/python3/dist-packages/UpdateManager/backend/ PyGIWarning: Gtk was imported without specifying a version first. Use gi.require_version(‘Gtk’, ‘3.0’) before import to ensure that the right version gets loaded.
from gi.repository import GLib, Gtk, Snapd
No protocol specified
Using TensorFlow backend.
2021-07-08 21:38:46.071513: I tensorflow/stream_executor/platform/default/] Successfully opened dynamic library
/usr/lib/python3.8/ MatplotlibDeprecationWarning:
The mpl_toolkits.axes_grid module was deprecated in Matplotlib 2.1 and will be removed two minor releases later. Use mpl_toolkits.axes_grid1 and mpl_toolkits.axisartist, which provide the same functionality instead.

(-c:12342): Gdk-CRITICAL **: 21:38:48.418: gdk_cursor_new_for_display: assertion ‘GDK_IS_DISPLAY (display)’ failed
/usr/lib/python3.8/ UserWarning: Viewer requires Qt
/usr/lib/python3/dist-packages/skcuda/ UserWarning: creating CUBLAS context to get version number
warnings.warn(‘creating CUBLAS context to get version number’)
[libprotobuf ERROR google/protobuf/] Invalid file descriptor data passed to EncodedDescriptorDatabase::Add().
[libprotobuf FATAL google/protobuf/] CHECK failed: GeneratedDatabase()->Add(encoded_file_descriptor, size):
terminate called after throwing an instance of ‘google::protobuf::FatalException’
what(): CHECK failed: GeneratedDatabase()->Add(encoded_file_descriptor, size):
*** Process received signal ***
Signal: Aborted (6)
Signal code: (-6)
[ 0] /lib/x86_64-linux-gnu/[0x7f871448f210]
[ 1] /lib/x86_64-linux-gnu/[0x7f871448f18b]
[ 2] /lib/x86_64-linux-gnu/[0x7f871446e859]
[ 3] /lib/x86_64-linux-gnu/[0x7f86c1708911]
[ 4] /lib/x86_64-linux-gnu/[0x7f86c171438c]
[ 5] /lib/x86_64-linux-gnu/[0x7f86c17143f7]
[ 6] /lib/x86_64-linux-gnu/[0x7f86c17146a9]
[ 7] /lib/x86_64-linux-gnu/[0x7f84b3922f7c]
[ 8] /lib/x86_64-linux-gnu/[0x7f84b397df1a]
[ 9] /lib/x86_64-linux-gnu/[0x7f84b3974dae]
[10] /lib/x86_64-linux-gnu/[0x7f871443847f]
[11] /lib/x86_64-linux-gnu/[0x7f84b3975b69]
[12] /lib64/[0x7f871466bb8a]
[13] /lib64/[0x7f871466bc91]
[14] /lib/x86_64-linux-gnu/[0x7f87145ac915]
[15] /lib64/[0x7f87146700bf]
[16] /lib/x86_64-linux-gnu/[0x7f87145ac8b8]
[17] /lib64/[0x7f871466f5fa]
[18] /lib/x86_64-linux-gnu/[0x7f871442134c]
[19] /lib/x86_64-linux-gnu/[0x7f87145ac8b8]
[20] /lib/x86_64-linux-gnu/[0x7f87145ac983]
[21] /lib/x86_64-linux-gnu/[0x7f8714421b59]
[22] /lib/x86_64-linux-gnu/[0x7f87144213da]
[23] python(_PyImport_FindSharedFuncptr+0x6f)[0x6737cf]
[24] python(_PyImport_LoadDynamicModuleWithSpec+0x15b)[0x6840db]
[25] python[0x684c71]
[26] python[0x5c3cb0]
[27] python(PyVectorcall_Call+0x58)[0x5f2168]
[28] python(_PyEval_EvalFrameDefault+0x6552)[0x570902]
[29] python(_PyEval_EvalCodeWithName+0x26a)[0x568d9a]
*** End of error message ***

Running the following from the bash terminal

python3 -c “import torch;print(torch.version);print(torch.cuda.is_available());print(torch.cuda.get_device_name(torch.cuda.current_device()));”


Tesla T4

I tried upgrading the lambda-stack with

sudo apt-get update && sudo apt-get dist-upgrade

and it seemed to work and printed no errors.

However, I keep getting the same failure on help(:modules").

How can I get a list of python modules from within python with lambda-stack?

Is this the right place to report this? Is there a git repo I should attempt to report it on?

Sorry for the delay.

  1. I confirmed the help(modules) works with the clean lambda stack.
    python -c "help('modules')"
  2. It seems you have a different iPython that is deprecated. And then there are Google protocol buffer differences.
    • So it looks like you have a mix of old software.
  3. To remove/clean up lambda stack, but also other old python packages, you can do the following:

The following will remove all NVidia software, deep learning libraries, and then
reinstall Lambda Stack.

sudo rm -f /etc/apt/sources.list.d/{graphics,nvidia,cuda}*; \
	COLUMNS=200 dpkg -l |
	awk '/cuda|lib(accinj64|cu(blas|dart|dnn|fft|inj|pti|rand|solver|sparse)|magma|nccl|npp|nv[^p])|nv(idia|ml)|tensor(flow|board)|torch/ { print $2 }' |
	sudo xargs -or apt -y remove --purge
    # Then it reinstalls Lambda Stack
    LAMBDA_REPO=$(mktemp) && \
	wget -O${LAMBDA_REPO} && \
	sudo dpkg -i ${LAMBDA_REPO} && rm -f ${LAMBDA_REPO} && \
	sudo apt-get -y update && sudo apt-get -y install lambda-stack-cuda

If that does not help, I can help you work through this.