Upgrading to later version of Python

[tancs711]

I need Python 3.9 or higher. The server I launch is 3.8 I ran

sudo apt-get update && sudo apt-get upgrade
sudo apt-get install python3.11

But still python3 --version shill shows 3.8. And my code did not run.

ValidationError: 1 validation error for PythonAstREPLTool
root
This tool relies on Python 3.9 or higher (as it uses new functionality in the ast module, you have Python version: 3.8.10 (default, May 26 2023, 14:05:08)
[GCC 9.4.0] (type=value_error)

[addavox]

This article should give you everything you need:
https://towardsdatascience.com/installing-multiple-alternative-versions-of-python-on-ubuntu-20-04-237be5177474

Typically, with python projects, you run your project in a virtual environment in which you can specify which interpreter you will use. That is a little bit longer discussion. The article above should get you rolling.

[tancs711]

Thanks @addavox. Appreciate this. I am new to python environment.

[tancs711]

I tried these and managed to installed another version on python. How do I get my jupyter notebook to use this version when I click on the launch link at the instance panel. The following sys.version still show that Jupyter notebook is linked to the 3.8 (although on jupyter terminal it shows 3.11).

I also create a new sudo user and created another env. And installed jupyter lab there. When I tunnel in to that, it still launch the 3.8 version.

[mpapili]

What you’re looking for is doable in the Web JupyterLab that Lambda sets up for you.

Here is a quick demonstration:

Step1: Setting up a normal instance, nothing special. 1x_A10 if curious, but not important for this demonstration:

Step2: Installing Python3.11 and setting up our VirtualEnv

After either SSH’ing in or using the Terminal sessions from the JupyerLab:

# install Python 3.11 to our system
sudo apt update
sudo apt install python3.11 python3.11-venv

now let’s set up the Virtual Environment and activate it normally:

# make the venv
python3.11 -m venv mikesVenv
# activate
source mikesVenv/bin/activate
# check
which python
/home/ubuntu/mikesVenv/bin/python

-…obviously, name the virtualenv whatever you’d like

Step3: Install ipykernel and add to Jupyter

Make sure your virtualenv is still activated from above and:

# install the ipykernel module
pip install ipykernel

# Add this environment to Jupyter
python -m ipykernel install --user --name=py311

Step4: Test it out!

Refresh your JupyterLab session for this instance and you should see your new py311 session ready and available to use if you go to File → New Launcher:

-…and just replicating your test, making sure that we are indeed using our Python3.11 install:

And that’s all there is to it! Let me know if this solution works for you.

One last note/disclaimer: This is a fresh Python environment, so you won’t have things like Torch and Tensorflow which Lambda sets up for you out the box until you pip install them yourself in the venv.

[tancs711]

@mpapili. Good news! I managed to change the kernel to 3.11

Can you let me know why the following did not work

• Note is using the 311 kernel
• Run !pip install openai.
• But when I try and import openai I receive the following.

---------------------------------------------------------------------------
ModuleNotFoundError                       Traceback (most recent call last)
Cell In[3], line 3
      1 get_ipython().system('pip -q install openai')
      2 import os
----> 3 import openai
      5 # This this to the location of the key file
      6 with open('keys.txt', 'r') as file:

ModuleNotFoundError: No module named 'openai'

[tancs711]

It seem I cannot execute !pip install openai from the notebook even when the kernel is 311 and expect the module to be installed to the 311 environment. Seem I need to run pip install open in a terminal with 311 myenv. Is this correct?

I am just wondering why previously I was able to !pip install openai from the notebook and it will be done to the native 3.8 version.

[mpapili]

While I’m not much of a Jupyter user personally, it seems like get_ipython().system() does not respect your current virtual environment in the same way that you may expect. It’s not running a session with the virtual environment “activated” in the same way (meaning $PATH’s are not set to point to your Python3.11’s “pip” and what you’re finding is the system’s “pip”).

Quick Demonstration:

Notice how my Python3.11 session is seeing the system pip while virtual environment’s pip (which will install modules in the desired PythonPath) is different.

Fortunately, you can still install modules from within your Jupyter session without needing to hop back and forth between the terminal. While the commands issued to the system through ! aren’t looking at your virtual environment, you can just use the full path to your Python interpreter:

!/home/ubuntu/mikesVenv/bin/python -m pip install openai

then…

import openai
openai.__file__

# returns '/home/ubuntu/mikesVenv/lib/python3.11/site-packages/openai/__init__.py'

as we can see, the install and import works in our Python3.11 session and the module is installed in our desired Virtual Environment without needing to hop back and forth between an SSH or Terminal session and your notebook.

Quick screenshot demonstrating this with requests and pandas:

[tancs711]

@mpapili,. Thanks. It all working now with your suggestion to prepend the path to the pip. It’s a little cumbersome if we change the environment. But importantly, it’s a solution so I can continue the R&D work.

[ctjlewis]

Not technically stable since you have to uninstall system packages but this should help. I needed this for use with axolotl.

https://gist.github.com/ctjlewis/7540d88f4ddb93d36e7515fb1b911833