Using QGIS processing algorithms outside the QGIS Python environment

Question

I'm very naively wondering how one can use any of the available native (C++) processing algorithms in a "QGIS-independent" Python Console.

More precisely, in the context of a bare system such as a Docker container where no GUI is available, what could be the minimal amount of package(s) to install to be able to import the processing algorithm from a standard Python console?

I've tried to install libqgis-core3.22.16 but this is fetching half the Universe, apt telling me that:

After this operation, 571 MB of additional disk space will be used.

on a bare Ubuntu 22.04 system.

After that, installing python3-qgis, the Python bindings to QGIS, is adding taking much more space:

After this operation, 790 MB of additional disk space will be used.

And even after that:

from qgis.core import *
Traceback (most recent call last):
 File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'qgis'

But at least, they seem to be present in the system:

# ls -l /usr/share/qgis/python/plugins/processing
total 64K
-rw-r--r-- 1 root root 400 Feb 3 2023 metadata.txt
-rw-r--r-- 1 root root 2.0K Feb 3 2023 __init__.py
-rw-r--r-- 1 root root 15K Feb 3 2023 ProcessingPlugin.py
drwxr-xr-x 5 root root 4.0K Sep 14 10:27 algs
drwxr-xr-x 2 root root 4.0K Sep 14 10:27 script
drwxr-xr-x 2 root root 4.0K Sep 14 10:27 preconfigured
drwxr-xr-x 2 root root 4.0K Sep 14 10:27 modeler
drwxr-xr-x 3 root root 4.0K Sep 14 10:27 images
drwxr-xr-x 2 root root 4.0K Sep 14 10:27 gui
drwxr-xr-x 2 root root 4.0K Sep 14 10:27 core
drwxr-xr-x 2 root root 4.0K Sep 14 10:27 ui
drwxr-xr-x 2 root root 4.0K Sep 14 10:27 tools
drwxr-xr-x 3 root root 4.0K Sep 14 10:27 tests

The final idea is to be able to call the same code as one would do in the QGIS Python environment for executing a processing algorithm:

import *necessary packages*
processing.run("native:buffer", {
 'INPUT': '/data/lines.shp',
 'DISTANCE': 100.0,
 'SEGMENTS': 10,
 'DISSOLVE': True,
 'END_CAP_STYLE': 0,
 'JOIN_STYLE': 0,
 'MITER_LIMIT': 10,
 'OUTPUT': '/data/buffers.shp'
})

I could have written down that question in another way: is it possible to port or convert any native QGIS processing algorithm (native:buffer) to Python? Even a single one of interest.

I'm aware that when executing any algorithm from the QGIS Python Console, the code needs to be aware of the iface and all the layers, etc. But in my case, I only care about processing files on disk so it's OK to lose all these internal QGIS objects.

• Matt: The geometry operations in QGIS are performed using the GEOS (C / C++) library. Shapely is built on the same backend. Is it an option to install and use Shapely instead?
• swiss_knight: Yes of course, if that means that all QGIS native processing algorithms would be made available through Shapely.

Answer 1

You've got 3 questions in there:

1. using native processing algs in a standalone python console
2. minimal amount of package(s) to install
3. porting native algs to python

I'll answer the first two.

The package python3-qgis requires a full QGIS install even if there's no gui available.

It's not well (or not obviously) documented how to use QGIS processing in a standalone script. I get the feeling that it's more intended for use within the GUI where all the setup is taken care of for you or using qgis_process from the commandline. But it can be done.

Note: I couldn't reproduce your ModuleNotFoundError: No module named 'qgis' in an Ubuntu 22.04 system (fresh VM, using QGIS.org packages). I didn't attempt a docker setup.

Using PyQGIS outside QGIS itself has some documentation which you do need to follow - "Using PyQGIS in standalone scripts".

I couldn't find any documentation, but to import processing, you need to tell Python where the module is either by adding it to your PYTHONPATH environment variable or using sys.path.append in your script.

Now you can import processing but you can't really use it yet as none of the native algorithms are loaded... Again, I couldn't find any documentation, but based on this GIS SE answer, you need to load the QgsNativeAlgorithms provider.

One last thing, you may need to set the environment variable QT_QPA_PLATFORM=offscreen to run without a display.

A working example:

import sys
from qgis.core import QgsApplication
from qgis.analysis import QgsNativeAlgorithms
# Tell qgis where it is installed, why this is necessary... I don't understand
QgsApplication.setPrefixPath(sys.prefix, True)
# Tell python where the processing package is
sys.path.append(f"{sys.prefix}/share/qgis/python/plugins")
# importing processing after PrefixPath is set avoids
# "Application path not initialized" warning in stdout
import processing
# Initialise QgsApplication. Setting the
# second argument to False disables the GUI.
qgs = QgsApplication([], False)
qgs.initQgis()
# Load Native Algorithm provider
QgsApplication.processingRegistry().addProvider(QgsNativeAlgorithms())
# Now finally we can run.....
processing.run("native:buffer", {
 'INPUT': 'lines.shp',
 'DISTANCE': 100.0,
 'SEGMENTS': 10,
 'DISSOLVE': True,
 'END_CAP_STYLE': 0,
 'JOIN_STYLE': 0,
 'MITER_LIMIT': 10,
 'OUTPUT': 'buffers.shp'
})