Adding missing python packages to QGIS Plugin?

Question

In our plugin, we dependent on some external python packages. Some are available in QGIS, some (like pandas or fiona) are not.

I've written small script, which checks requirements and if something is missing, it is automatically installed.

install_deps.py:

import pathlib
import sys
plugin_dir = pathlib.Path(__file__).parent.parent
try:
 import pip
except ImportError:
 exec(
 open(str(pathlib.Path(plugin_dir, 'scripts', 'get_pip.py'))).read()
 )
 import pip
 # just in case the included version is old
 pip.main(['install', '--upgrade', 'pip'])
sys.path.append(plugin_dir)
with open(str(plugin_dir / 'requirements.txt'), "r") as requirements:
 for dep in requirements.readlines():
 dep = dep.strip().split("==")[0]
 try:
 __import__(dep)
 except ImportError as e:
 print("{} not available, installing".format(dep))
 pip.main(['install', dep])

In the main file of python plugin, I just have import install_deps and missing modules are pip-installed automatically.

Disadvantage is, that this short "check" is done every time QGIS is starting (with the plugin activate).

Not sure, if there is better approach, than this?

• Fran Raga: where is this code?__init__ plugin?check this sample github.com/All4Gis/QGISFMV/blob/master/code/__init__.py#L18
• Jachym: My code is github.com/OpenGeoLabs/qgis-ndop-downloader/blob/deps/… but your approach is more or less the same. Still, I would prefer some qgis-based dependency management system than just hack like this. But I'm only asking, whether I'm not missing something.
• Fran Raga: there is no dependency management of own qgis

Answer 1

Don't do this.

It would be super irritating if some QGIS plugin would install/upgrade python packages behind the user's back. Please don't do this unless you isolate this 100% to just your own plugin, which is either impossible or incredibly complex.

You risk breaking other plugins, breaking other software, breaking the user's system!

Imagine if another plugin always installed its own specific pandas version that is incompatible with yours.

Instead:

• make a prominent note about which packages in which versions are needed
• make your plugin try and fail gracefully
• let the user themselves handle external dependencies

• Jachym: While I see the problem of doing something in hidden, I'm facing the issue, that plugin users are not even familiar with pip and they are using windows with no idea about what is python and why there are some dependencies needed. This is the problem I need to address in some user-friendly way. One of the options would be to install missing packages into the plugin directory - that would prevent rest of the system from being contaminated.
• bugmenot123: I totally get the problem, it is hard :(( Installing just locally to the plugin directory might work, nice idea! But you mentioned Fiona which requires GDAL and that is a whole other can of worms of (DLL/library) complexity to install and load properly on Windows, Linux and MacOS... If you can drop Fiona in favour of QGIS' built-in I/O functionality, I would highly recommend that. About pandas I know nothing, if it is plain python files, maybe bundle it right into your plugin?
• Jachym: Well I try to write the plugins in platform-independent way, so that one script can be used on the command line as well as part of the plugin. It would probably be possible to implement the functionality with core QGIS tools, or - for the worst scenario - with python-gdal (which I believe is part of OSGeo4W package(?)). But fiona (pandas is another layer on top of FIona) does the job in more nicely way. Well, thanks for your input anyway. I just wanted to be sure, there is no official optimal solution (except for writing documentation and educate users).
• bugmenot123: I would suggest bringing this up on lists.osgeo.org/listinfo/qgis-developer

Answer 2

Like bugmenot123 said, don't do this behind the user's back.

Let the code check wether a file is present or not? When it is present the code has never been run. When it has finished running the first time it will delete the (rename) the install_deps.py file.

import os
if os.path.isfile('install_deps.py'):
 print('WARNING: new dependies will be installed....')
 import install_deps
 install_deps.installer_func
 os.rename('install_deps.py', 'install_deps.installed')
else:
 pass

Place youre code into a function:

import pathlib
import sys
def installer_func:
 # youre code from install_deps.py

• Jachym: While I understand the argument about not to install anything behind users's back, and I probably understand your code @BearHunt, your proposal seems little bit in contradiction. I like the idea of renaming the file after deps are installed. But it's still happening behind users back - right? I can come with some graphical dialog for this, so it's transparent ... but that requires again bunch of coding :-(
• BearHunt: Well, i warn the user that the deps are going to be installed. Furtermore my answer was not meant as a solution for that problem, only for the 'first runtime'.

Answer 3

The place were the blessed route to a solution will be, if it can be found, is PIP dependencies for Python plugins QGIS enhancement proposal. It's a hard problem to solve. That thread contains exploration of what doesn't work well enough and some interim approaches.

The cleanest to my eye is https://github.com/qgis/QGIS-Enhancement-Proposals/issues/202#issuecomment-815844271, an eye that has read throught things but not tried to implement anything! So do read and think through the rest.

...we solved the problem of conflicting versions of libraries with our plugins when I was working for Boundles was to alter the path through sites and ship all the dependencies within the plugin itself.

I can't remember the details @volaya knows probably more. Here is an example of a plugin that was doing that: https://github.com/planetfederal/qgis-geoserver-plugin/blob/master/geoserverexplorer/extlibs/site.py

So, each plugin imported its dependencies from a module plugin_namespace.extlibs actually insulating them from other plugins. IIRC it was working well but I can't remember if it led to issues with binaries.