How to configure QGIS on Ubuntu to use a virtual environment for custom Python libraries

Question

I am trying to figure out the best way to run QGIS with custom python libraries and I gather I should create a virtual environment, but I am open to other suggestions. Currently, I am trying to configure QGIS 3.40.3 (Bratislava) on Ubuntu (tuxedo-os specifically) to use a custom Python environment so I can install and use libraries like hdbscan, which are only available via pip. I want to avoid installing packages system-wide to prevent breaking dependencies.

Steps I Took:

Created a Virtual Environment

python3 -m venv ~/qgis-venv

Then, activated it:

source ~/qgis-venv/bin/activate

Installed hdbscan and Dependencies

pip install --upgrade pip 
pip install hdbscan numpy scipy

Set PYTHONPATH in QGIS Environment Settings

Opened QGIS → Settings → Options → System → Environment Added: PYTHONPATH = /home/user/qgis-venv/lib/python3.12/site-packages

After restarting I get the error on startup and python suport is disabled.

Couldn't load PyQGIS. Python support will be disabled. Traceback (most recent call last): File "", line 1, in File "/usr/lib/python3/dist-packages/qgis/core/init.py", line 22822, in

import shapely as _shapely File "/usr/lib/python3/dist-packages/shapely/__init__.py", line 1, in 
from shapely.lib import GEOSException # NOQA
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ImportError: numpy.core.multiarray failed to import

Next I tried running specifying PYTHONPATH in bash

PYTHONPATH=~/qgis-venv/lib/python3.12/site-packages qgis

and I get the same error.

Alternatively, I Tried Running QGIS from the Terminal with the Virtual Environment

(qgis-venv) user@user-tuxedo-os:~$ qgis

This led to the following error

Couldn't load SIP module. Python support will be disabled.

Traceback (most recent call last): File "", line 1, in ModuleNotFoundError: No module named 'qgis'

What I Need Help With:

1. What is the best way to configure QGIS to use a virtual environment on Ubuntu/Tuxedo without breaking system dependencies?
2. Should I be using PYTHONPATH, PYTHONHOME, or another approach?
3. How can I ensure QGIS finds its required modules (sip, qgis) while still using my custom environment for additional packages?

Also, I am open to other suggestions. Maybe I am going about this all wrong, because it feels like this has been way more trouble to get one library loaded then I would have anticipated coming from R.

• Cannon: I found this answer gis.stackexchange.com/questions/403078/… which requires installing anaconda and installing qgis directly from anaconda. I saw on a thread on github that installing qgis from anaconda has many known issues so I was trying to avoid this and go the venv route. Maybe this is not possible...

Answer 1

I have found a solution that requires starting up QGIS normally (not in a virtual environment). Then I run the following script

import sys
sys.path.append("/home/user/qgis-venv/lib/python3.12/site-packages")

Then I can import hdbscan without errors

import hdbscan

For the meantime I will just add similar code to the top of my python scripts

import sys
venv_path = "/home/user/qgis-venv/lib/python3.12/site-packages"
if venv_path not in sys.path:
 sys.path.append(venv_path)
 

which checks to see if file path is already there before appending, so the list doesn't grow long when running scripts multiple times. I think this solution works well for me, because I would much rather edit my scripts than my settings between sessions. But if you come across this post and see a better solution, please let me know.

Also, the strategy to set PYTHONPATH in the QGIS Environment Settings would have worked if I had used all the rights paths in the value field:

/usr/share/qgis/python:/usr/lib/python3/dist-packages:/home/user/qgis-venv/lib/python3.12/site-packages

This approach is fine but it will also permanently include the venv in your system paths until you edit your settings again. That might be preferable for some users.