Saving Python objects into memory for repeated use of Python toolbox?

Question

I have made a Python toolbox which processes relatively large polyline feature class (millions of features), builds a network graph from it as dictionary of lists, and depending on user selected features in map finds shortest paths between selected features and creates a new selection of these shortest paths. This tool is used repeatedly in one of our workflows. I would love to keep the "network graph" as dictionary of lists in memory. I am aware of in_memory workspace and possibility to convert my dictionary into table and store it like that there.

I wonder if there is some way to save directly Python objects into generally accessible space (variable) in memory in ArcGIS Pro from Python toolbox?

Something like in_memory workspace but directly for Python objects (variables). In my particular case so I can check if this dictionary of lists exists and potentially reuse it if object meets the user settings and other requirements, when the tool is rerun?

• Son of a Beach: You could do this in ArcMap using Python Add-Ins, which can support persistent objects in memory. However, Python Add-Ins are not available in ArcGIS Pro. At least not yet. Vote it up at: community.esri.com/ideas/10509-python-add-ins-for-arcgis-pro
• Miro: @SonofaBeach That is actually exactly what I did in ArcMap, but now we are slowly migrating into ArcGIS Pro and I would like to have some solution for that.
• PolyGeo: I wonder whether this may be a Python (Stack Overflow) rather than ArcPy (GIS SE) topic: stackoverflow.com/questions/6687660/…
• Miro: @PolyGeo Thank you, some answers there seems they could work. I will give it a go and report back.
• Fran Raga: maybe using some python cache library like github.com/pydanny/cached-property or github.com/tkem/cachetools

Answer 1

In VB .net if you want to save an object to file to be used later, e.g. a dictionary you would serialize it. I believe the equivalent in python is pickle.

Be aware not all objects can be "pickled", I would imagine especially anything arcpy related (but I may be wrong).

So if you are extracting network topology as dictionaries then you should be able to pickle them and then reload them back into memory from a well known location at a later date.

My experience with serialization is principally in VB. net so I can't comment on the speed of pickling or the reading back into memory but I would imagine it should be quite fast.

Answer 2

Just some thoughts about working with graphs in Python. I believe the base ArcGIS Pro and Desktop both come loaded with Numpy and Scipy, so these should be available to everyone. You could consider using a sparse matrix instead of a dictionary of lists. These will use less memory and be faster computing. You can also perform djikstra's shortest path algorithms using Scipy. You can save the sparse matrix and load it as needed. The matrix is held entirely in memory, which may be a problem for you depending on how large the files are.

Creating graphs (also has links to different shortest path algorithms):

https://docs.scipy.org/doc/scipy-0.16.1/reference/sparse.csgraph.html

Basic shortest path algorithm:

https://docs.scipy.org/doc/scipy-0.16.1/reference/sparse.csgraph.html

To save the sparse matrix:

https://docs.scipy.org/doc/scipy/reference/generated/scipy.sparse.save_npz.html

• Miro: Thank you, I do use scipy for some parts and implementation of Dijkstra using heapq heap which works really well with dictionary of lists.