Issues with module imports going from python 2 to python 3

Question

I am trying to upgrade a 10 year old event listener that I didn't write from Python 2.7 to python 3.7. The basic issue I'm running into is the way the original script was importing its plugins. The idea behind the original script was that any python file put into a "plugins" folder, with a "registerCallbacks" function inside it would auto-load itself into the event listener and run. It's been working great for lots of studios for years, but Python 3.7 is not liking it at all.
The folder structure for the original code is as follows:

EventListenerPackage
 src
 event_listener.py
 plugins
 plugin_1.py
 plugin_2.py

From this, you can see that both the event listener and the plugins are held in folders that are parallel to each other, not nested.
The original code read like this:

# Python 2.7 implementation
import imp
class Plugin(object):
 def __init__(self, path):
 self._path = 'c:/full/path/to/EventListenerPackage/plugins/plugin_1.py'
 self._pluginName = 'plugin_1'
 def load(self):
 try:
 plugin = imp.load_source(self._pluginName, self._path)
 except:
 self._active = False
 self.logger.error('Could not load the plugin at %s.\n\n%s', self._path, traceback.format_exc())
 return
 regFunc = getattr(plugin, 'registerCallbacks', None)

Due to the nature of the changes (as I understand them) in the way that Python 3 imports modules, none of the other message boards seem to be getting me to the answer.
I have tried several different approaches, the best so far being:
How to import a module given the full path?
I've tried several different methods, including adding the full path to the sys.path, but I always get "ModuleNotFoundError".
Here is roughly where I'm at now.

import importlib.util
import importlib.abc
import importlib
class Plugin(object):
 def __init__(self, path):
 self._path = 'c:/full/path/to/EventListenerPackage/plugins/plugin_1.py'
 self._pluginName = 'plugin_1'
 def load(self):
 try:
 spec = importlib.util.spec_from_file_location('plugins.%s' % self._pluginName, self._path)
 plugin = importlib.util.module_from_spec(spec)
 # OR I HAVE ALSO TRIED
 plugin = importlib.import_module(self._path)
 except:
 self._active = False
 self.logger.error('Could not load the plugin at %s.\n\n%s', self._path, traceback.format_exc())
 return
 regFunc = getattr(plugin, 'registerCallbacks', None)

Does anyone have any insights into how I can actually import these modules with the given folder structure? Thanks in advance.

• Geoduck: looked at importlib.machinery.SourceFileLoader?
• Frank Merrow: Are you sure the modules are valid for Python3? Also, some other hints and suggestion might be found here: stackoverflow.com/questions/43728431/…

Answer 1

You're treating plugins like it's a package. It's not. It's just a folder you happen to have your plugin source code in.

You need to stop putting plugins. in front of the module name argument in spec_from_file_location:

spec = importlib.util.spec_from_file_location(self._pluginName, self._path)

Aside from that, you're also missing the part that actually executes the module's code:

spec.loader.exec_module(plugin)

---

Depending on how you want your plugin system to interact with regular modules, you could alternatively just stick the plugin directory onto the import path:

sys.path.append(plugin_directory)

and then import your plugins with import or importlib.import_module. Probably importlib.import_module, since it sounds like the plugin loader won't know plugin names in advance:

plugin = importlib.import_module(plugin_name)

If you do this, plugins will be treated as ordinary modules, with consequences like not being able to safely pick a plugin name that collides with an installed module.

---

As an entirely separate issue, it's pretty weird that your Plugin class completely ignores its path argument.

• Adam Benson: Yeah, I'm not sure why the plugin path is getting ignored. It's frustrating, but the information you all have provided so far has been very helpful. The spec.loader.exec_module(plugin) I have now added, though I may have to figure out where in the original script those things are being executed. Does spec.loader.exec_module(plugin) execute it right away, or does it just prepare it to be used elsewhere? Thanks again. This has been very helpful
• user2357112: @AdamBenson: spec.loader.exec_module(plugin) executes it right away.