I often use the line magics

%reload_ext autoreload
%autoreload 2

At the start of the notebook to allow for incremental development of a library. Maybe that will work on this case?

Make sure the pip you're using is the one inside the notebook's environment. You can try running this in a notebook cell:

!which pip
!which python

If pip is outside the environment, you'll need to figure out how to activate the environment before running pip install.

Another way to solve this is to use sys.executable and subprocess:

import sys
import subprocess
subprocess.check_call([sys.executable, '-m', 'pip', 'install', 'numpy==1.21'])
import numpy as np
print(np.__version__)

This code directly calls pip using the same Python interpreter that is currently running. After the installation, NumPy is imported again, and the version is printed to verify the installation.

The problem is Jupyter notebook kernel cache the initial loaded version of a package and because of it !pip install numpy==1.21 won't work.

Try this

!{sys.executable} -m pip install numpy==1.21

I am not sure what "normal way" is not working, but understanding virtual environments and versioning is essential to understanding this.

Language Example: Py3 vs Py2

As an example, this question is tagged as Python 3.x which is a major update from Python 2.x including breaking changes. Updates to the print function are an important breaking change and highlights the issue.

This works in Py 2.x:

print "hello"

Py 3.x requires:

print("hello")

So Py 2.x code often fails in Py 3.x (often because not all of the language changes, running 2+2 would work in both). However, updating all Py 2.x code is not practical or necessary. Instead, we use environments to allow multiple installations with difference language and package versions.

Package Versioning

Packages work in much the same way where versions may have bug fixes, updates, or breaking changes across versions. So functionality differs between versions. When working on multiple projects, switching/updating/downgrading package versions for each project is a headache. If Project 1 requires numpy==1.26 and Project 2 numpy==1.21, switching is hard work in a single environment where you have to run a pip command (or many more) when switching. Package compatibility issues will crop up whenever switching between if all the proper changes are not made.

Dependency Issues

Python has an extensive set of packages to extend functionality and they often rely or depend on other packages. In the example script numpy is required to run, specifically numpy==1.21. numpy is a cornerstone of many, many other python packages including pandas, matplotlib, and sklearn. Tools like pip help ensure that the right packages and their dependencies are installed together.

Switching between numpy versions can have repercussions for other packages and break functionality.

Switching versions mid-workflow = more issues or unpredictable behavior.

Package and Environment Managers

Keeping track of all this, can be hard. Which is why package and environment managers like pip, conda, and venv are used. They help ensure everything can work together. The below examples focus on installing numpy==1.21 with Python 3.9x (which I think is the last compatible version for this np version).

pip

Python's default package manager.

Basic usage pip install numpy. Allows for version pinning such as pip install numpy==1.21. Will try to resolve any dependency issues with installation. NOTE: Downgrading a package like numpy may necessitate downgrades to other packages.

venv

Creates virtual environments where different versions of Python and packages can be used. This allows for, and is recommended for, appropriate (e.g., Py3 vs Py2), isolated (just the packages you need), and versioning (the right versions and dependencies) across multiple projects.

This approach requires the desired version of Python to be installed (for example testing with Python3.12 failed with this numpy version, 3.9 worked).

See Python Download Org Official or use HomeBrew on Mac or Anaconda as preferred.

Check Python Version and Install venv

python --version
pip install virtualenv

Create and Activate New Environment

This will create a new folder where it is run called myvenv_py3. Activate on Mac/Unix systems (may be different on Windows). Use pip to install and upgrade required packages. Create the new environment as a JupyterLab kernel.

python3.9 -m venv myvenv_py3
source myvenv_py3/bin/activate
pip install --upgrade pip
pip install numpy==1.21
pip install ipykernel
python -m ipykernel install --user --name=myvenv_py39 --display-name "My Venv Python 3.9 (numpy 1.21)"

conda Alternative

Conda is another popular package and environment manager for Python. Below is a mirrored workflow for using it.

# Create a conda environment with Python 3.9
conda create -n myconda_py39 python=3.9
# Activate the environment
conda activate myconda_py39
# Install NumPy 1.21
conda install numpy=1.21
# Install ipykernel to use in JupyterLab
conda install ipykernel
# Register this environment as a Jupyter kernel
python -m ipykernel install --user --name=myconda_py39 --display-name "My Conda Python 3.9 (numpy 1.21)"

Jupyter Lab Kernels

Jupyter Lab now shows the newly added kernels with the correct associated packages.

Jupyter Kernel Options

Note: Jupyter lab only needs to be installed in one environment once kernels are added. But be sure to run jupyter lab from that environment and then select correct kernel.

TLDR

Trying to change an actively used package in Python is tricky. Jupyter uses a kind of "active memory" when running, hence indicating a restart is required when changing numpy versions. I find the reload tips handy when working on my own imports (e.g., my_functions.py) where I am doing updates and want to test. It is not a stable solution for larger packages that are dependencies of currently running processes.

The right way to handle these requirements is with multiple environments.

• python
• python-3.x
• linux
• numpy
• sys