Using Virtual Environments with Lambda Stack

[Johann]

I am using the Tensorbook, which comes with the preinstalled libraries as part of the Lambda Stack, but I don’t want to be developing in the root environment and want to create a new Virtual Environment.

Is there a best practices for this, or commands, or recommended process to get the Lambda Stack for virtual environments?

[sabalaba]

We’re still establishing the best practices. There are two main choices I see:

1. Use Lambda Stack as an easy way to install your Drivers, CUDA, CuDNN, etc.
2. Use Lambda Stack’s version of TensorFlow / PyTorch.

If you decide to go for type 1, you’ll simply install Lambda Stack and then create your virtualenv like this:

virtualenv -p python3 your_venv

If you decide to go for type 2, you’ll install Lambda Stack and then create your virtualenv like this:

virtualenv --system-site-packages -p python3 your_venv

Essentially in type 2 you’ll be using our TensorFlow / PyTorch packages and type 1 you’ll be installing them yourself.

[msuttles]

Since this is from about a year ago, I wondered if you could update if there are now best practices for virtual environments? Thanks.

[saurabh]

Are there any updates on this?

[rneely]

@sabalaba Thank you. If we use the 2nd option:
virtualenv --system-site-packages -p python3 your_venv
and then install additional packages from an additional requiremnts.txt file
would the additional packages be installed in the venv, or where system site packages are installed?

I have a lambda-labs tensor book.
We want to use Data Science cookie cutter for our project: https://drivendata.github.io/cookiecutter-data-science/ which suggests, as a best practice, to have a separate python venv for each project.

[rajatalak]

Is there a way to do this (option 2) using IDE like PyCharm?

[markd]

@rajatalak ajatalak

Here is the two ways that PyCharm seems to support VirtualEnv:

1. It can use a existing virtualenv
2. It can create a virtualenv

Documents at:
https://www.jetbrains.com/help/pycharm/creating-virtual-environment.html#python_create_virtual_env

Mark

[wgiersche]

Is there any way to achieve the same with pipenv?

[markd]

It should be pretty much the same. (but below are the ones we have written up)

We have documented this for:
- Docker - https://lambdalabs.com/blog/nvidia-ngc-tutorial-run-pytorch-docker-container-using-nvidia-container-toolkit-on-ubuntu/
- python venv - https://lambdalabs.com/blog/p/120e0f36-3dc2-4164-be0d-9c80705017e7/
- virtualenv - https://lambdalabs.com/blog/p/8060fba4-4ab4-464c-a829-d8fc917fcb1a/
- Anaconda/miniconda - https://lambdalabs.com/blog/setting-up-a-anaconda-environment/
(I missed the step in this that I installed cuDNN or you could use the anaconda path).
* Anaconda is missing a step of setting the LD_LIBRARY_PATH for its cuDNN so that
needs to be manually done. (The following will fix that):
export LD_LIBRARY_PATH=${CONDA_PREFIX}/lib:${LD_LIBRARY_PATH}

I hope that helps