2

I have a Python function that downloads a geopackage from somewhere then does some stuff with it and at some point converts it to an ESRI file geodatabase.
Now inside a python script I want to use this method to download and convert multiple geopackages at once. This is some stuff that in my eyes should be easy to run in parallel, since there is no overlap between input and output data of the different geopackages whatsoever.
So i tried implementing that, but somehow arcpy.management.FeatureClassToGeodatabase now raised the following error:
WARNING: C:\path\to\file.gpkg\lyr. ERROR 87931
which with the help of arcpy.GetIDMessage(87931) becomes 'Object: Tool or environment <%1> not found'
I was able to isolate this error and found out, that it has to do with the ThreadPoolExecutor. I created a minimal script which I used to reproduce this behaviour:

import asyncio
import concurrent.futures
import arcpy.conversion
import arcpy.management
async def main():
 # works
 await normal_test()
 # doesn't work
 await threadpool_test()
async def normal_test():
 convert()
async def threadpool_test():
 executor = concurrent.futures.ThreadPoolExecutor(4)
 await asyncio.get_event_loop().run_in_executor(executor, convert)
def convert():
 arcpy.management.CreateFileGDB("<path>", "out")
 arcpy.conversion.FeatureClassToGeodatabase("test.gpkg/lyr", "out.gdb")
 print("After converting")
# start whole script
asyncio.run(main())

So without even running something in parallel yet, the FeatureClassToGeodatabase fails when run inside a ThreadPoolExecutor, but works fine when called normally. CreateFileGDB however works in both scenarios.

Does anybody have an idea why some functions work inside a ThreadPoolExecutor while others do not and maybe knows a way to get them to work?

Environment:
I'm using ArcPy of ARCGIS Pro 2.9 and Python 3.7.11.
I'm working on Windows

Edit: Based on a comments recommendation i also tried to use the multiprocessing module without asyncio: (The convert method is the same as above)

import arcpy.conversion
import arcpy.management
from multiprocessing import Process
if __name__ == "__main__":
 proc = Process(target=convert)
 proc.start()
 proc.join()
 print("Finished")

This code has the same problem, the FeatureClassToGeodatabase just doesn't work, however this time there isn't even an error. The print statement "After converting" just gets executed after one second and nothing happened. Certainly no converting...

asked Jul 6, 2023 at 9:03
7
  • Sorry didn't realize my edit summary got clipped. Other than the title i removed the second question to make it a little bit more focused. Is it focused enough now? Commented Jul 6, 2023 at 12:09
  • Shot in the dark since I don't have experience with Pro, but in the past (Desktop) I've had to use multiprocessing.set_executable() to point at pythonw.exe, since sys.executable was being set to arcmap's exe. It would also simplify things to get rid of asyncio, at least for the minimal reproducible example. Commented Jul 7, 2023 at 21:49
  • ThreadPoolExecutor is different from multiprocessing. I used that because usually I have all sorts of problems with multiprocessing that just work right away with the ThreadPoolExecutor. But I tested it now with the multiprocessing module (Both using a ProcessPoolExecutor with asyncio and using process.start etc. directly). That won't produce an error, but somehow it doesn't work either. The FeatureClassToGeodatabase will just not do anything it seems. If i put a print statement after, it will get called after a second but normally this should take about 2-3 minutes... Commented Jul 10, 2023 at 9:11
  • @mikewatt I added the example code to my question. Commented Jul 10, 2023 at 9:33
  • You can use concurrent.futures without asyncio, was what I meant: docs.python.org/3/library/concurrent.futures.html Commented Jul 10, 2023 at 16:38

0

Know someone who can answer? Share a link to this question via email, Twitter, or Facebook.

Your Answer

Draft saved
Draft discarded

Sign up or log in

Sign up using Google
Sign up using Email and Password

Post as a guest

Required, but never shown

Post as a guest

Required, but never shown

By clicking "Post Your Answer", you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.