Sharing modules across python processes

Question

We are writing a system where memory is tight. There are multiple python processes that will import the same set of classes. If the processes load a gazillion classes and each process consumes a couple hundred megs, then 10 processes and I am running into gigs. Is there a way to "share" class imports across python processes?

Summary:

import classA # Process 1 : loads classA in memory
import classA # Process 2 : reuses what was loaded previously by Process 1

PS: What I'm trying to achieve is something that you would do with .so modules in C/C++.

• SingleNegationElimination: is there a reason you cannot use a dylib in your python exactly as you would in C or c++? You can load them pretty easily with the ctypes module, or if they are written as python extensions, by actually importing them.

Answer 1

It's possible to save at least a bit of memory if your OS supports an efficient copy-on-write fork (many do). Just import everything once in the parent, then fork() to create all the children.

Note that you won't save anything if you have many small objects, as the objects' refcounts will need to be written to. However, you could see substantial savings with large static structures like attribute dictionaries.

Answer 2

I don't believe this is possible in the design of python. You might want to uses threads instead of separate processes if this is a big deal. Also, if you're running 100 Megs just by importing several modules you might be doing something wrong in the modules (seems like quite a bit of memory to be using).

One possibility if you absolutely must import everything, can't cut down on memory usage and can't use threads would be to move the big memory code into python extensions written in C/C++. This would then allow you to share the common segments of the shared library your python extensions are in across processes. It also might reduce the memory footprint somewhat.

P.S. In python you're actually importing modules, not classes. Similar to how in C/C++ you're including header files, not classes.

• user395760: In C and C++, you're not importing anything. You're copying and pasting declarations automatically, and link compiled code together.
• CrazyCasta: @delnan Yes, just trying to point out that it's not the classes that are getting imported, it's modules. So you wouldn't be sharing class imports you'd be sharing module imports.
• user395760: Yes, and I approve that. I'm just prone to nitpicking :)
• gurshi: "One possibility if you absolutely must import everything, can't cut down on memory usage and can't use threads would be to move the big memory code into python extensions written in C/C++. This would then allow you to share the common segments of the shared library your python extensions are in across processes" -- I dont think this is true. If you import a native .so across python modules, you are still bringing them all in the scope of every module, and the import behavior as I understand copies each class in all the importer modules. Right?
• CrazyCasta: I was suggesting that you could put whatever data is constant (the stuff that you want to share across processes) in the shared modules as constant data. If you need to expose say an tuple of a million values, instead of actually exposing a tuple to python you would expose an object that overloads w/e the proper operator is (or implements a method, I'm fuzzy on the specifics) that then accesses the native array of values. This is just an example and you may need to tailor the basic idea for what you want.