imported module not found in class in Pyodide

Question

I am trying to use pyodide with lxml and urllib3, for some reasons I don't understand, when I try to use urllib3 in a class supposed to be a Resolver for lxml etree I get an error NameError: name 'urllib3' is not defined.

Example code is online at https://martin-honnen.github.io/pyodide-tests/simple-requests-test-case3.html (note that all output goes to the browser console so use F12 to see the output).

Code is doing <script src="https://cdn.jsdelivr.net/pyodide/v0.26.0/full/pyodide.js"></script> and then

 const python = `
import js
import urllib3
import lxml
from lxml import etree as ET
url = base_url + 'foo-transform-module.xsl'
js.console.log(urllib3.request('GET', url).status)
class TestResolver(ET.Resolver):
 def resolve(self, url, id, context):
 print("Resolving URL '%s'" % url)
 if url.startswith('http'):
 return self.resolve_file(urllib3.request('GET', url), context)
 else:
 return False
parser = ET.XMLParser(no_network=False)
parser.resolvers.add(TestResolver())
tree = ET.parse(url, parser)
tree.getroot().tag
 `;
 async function main() {
 let pyodide = await loadPyodide();
 await pyodide.loadPackagesFromImports(python);
 const locals = new Map();
 locals.set('base_url', window.location.href.replace(/[^/]+?$/, ''));
 console.log(await pyodide.runPythonAsync(python, { locals : locals }));
 };
 main();

Full error in console is

Uncaught (in promise) PythonError: Traceback (most recent call last):
 File "/lib/python312.zip/_pyodide/_base.py", line 574, in eval_code_async
 await CodeRunner(
 File "/lib/python312.zip/_pyodide/_base.py", line 394, in run_async
 coroutine = eval(self.code, globals, locals)
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "<exec>", line 28, in <module>
 File "src/lxml/etree.pyx", line 3570, in lxml.etree.parse
 File "src/lxml/parser.pxi", line 1952, in lxml.etree._parseDocument
 File "src/lxml/parser.pxi", line 1978, in lxml.etree._parseDocumentFromURL
 File "src/lxml/parser.pxi", line 1881, in lxml.etree._parseDocFromFile
 File "src/lxml/parser.pxi", line 1200, in lxml.etree._BaseParser._parseDocFromFile
 File "src/lxml/parser.pxi", line 633, in lxml.etree._ParserContext._handleParseResultDoc
 File "src/lxml/parser.pxi", line 739, in lxml.etree._handleParseResult
 File "src/lxml/etree.pyx", line 329, in lxml.etree._ExceptionContext._raise_if_stored
 File "src/lxml/parser.pxi", line 462, in lxml.etree._local_resolver
 File "src/lxml/docloader.pxi", line 150, in lxml.etree._ResolverRegistry.resolve
 File "<exec>", line 20, in resolve
NameError: name 'urllib3' is not defined
 at new_error (pyodide.asm.js:10:9965)
 at pyodide.asm.wasm:0x16dbeb
 at pyodide.asm.wasm:0x177339
 at _PyEM_TrampolineCall_JS (pyodide.asm.js:10:125866)
 at pyodide.asm.wasm:0x1c2db7
 at pyodide.asm.wasm:0x2c7b17
 at pyodide.asm.wasm:0x20a78c
 at pyodide.asm.wasm:0x1c34a4
 at pyodide.asm.wasm:0x1c37b3
 at pyodide.asm.wasm:0x1c3831
 at pyodide.asm.wasm:0x29e865
 at pyodide.asm.wasm:0x2a4e5c
 at pyodide.asm.wasm:0x1c3971
 at pyodide.asm.wasm:0x1c35da
 at pyodide.asm.wasm:0x17699d
 at callPyObjectKwargs (pyodide.asm.js:10:64068)
 at Module.callPyObjectMaybePromising (pyodide.asm.js:10:65316)
 at wrapper (pyodide.asm.js:10:27006)
 at onGlobalMessage (pyodide.asm.js:10:101760)

Is there some flaw in my code? How do I get the imported module to be known in the code of the class?

• Martin Honnen: Now filed as an issue at github.com/pyodide/pyodide/issues/4834.

Answer 1

I deleted old answer because it was wrong.

---

It seems all problem makes locals - maybe it removes original urllib3

I found Pyoide FAQ: How can I execute code in a custom namespace?.
It shows some examples with globals.

It seems this works for me:

let my_namespace = pyodide.toPy({'base_url': window.location.href.replace(/[^/]+?$/, '')});
 
await pyodide.runPythonAsync(python, {globals: my_namespace});

---

Information for other users: problem was send by OP to Pyodide as issue

imported module not found in class in pyodide, error "NameError: name 'urllib3' is not defined" · Issue #4834 · pyodide/pyodide

but there is already similar issue

Document weird behavior of runPython locals parameter · Issue #4673 · pyodide/pyodide

• Martin Honnen: Thanks for your investigation and suggestions, I tried to use the ordered and separate loadPackage approach you suggested in martin-honnen.github.io/pyodide-tests/… but there it doesn't help; on the other hand I have martin-honnen.github.io/pyodide-tests/… where I use loadPackagesImports and the Resolver works, finding urllib3, it appears. I will look into opening an issue on pyodide.
• furas: it seems problem makes locals - maybe it needs to get original locals and globals and send then to runPythonAsync
• Martin Honnen: I couldn't make sense of the different results, I have filed the issue github.com/pyodide/pyodide/issues/4834 today, let's see whether someone there can tell and explain whether it is a quirk or bug in pyodide and/or whether there is a workaround.
• furas: maybe when you set locals then it removes original locals and it lost access to urllib3 - but urllib3 should be rather in globals. At this moment only idea is to use f-string to put base_url directly in string with python code.
• furas: Pyoide FAQ: How can I execute code in a custom namespace? - it shows some examples with globals. Maybe it will help.