Does any inter-module shared memory based solution exist in python?

Question

I made a code in python using theano. Because theano uses GPU, hence I wanted to utilize the time it uses to computation with GPU for computing an other similarly computing time hungry process via CPU.

I worked a lot with multiprocessing module, but unfortunately I found, that theano doesn't work along with multiprocessing module resulting in some locking error. Thus I altered my code and wanted to run the theano based module (modul1.py) and the other one(modul2.py) along each other in two separate modules without multiprocessing.

Aftermath: I have to share data among separate modules. I tried pickling, but is too slow for my solution, I waste the time I would gain with "multiprocessing" because of the file operations.

Hence I need a solution, with which I can exchange data between modules inside of MEMORY.

Note: I need only simplex data exchange i.e. I want to read only from one of the modules.

Answer 1

Maybe mmap can help you , it is in standard library.

Documentation: https://docs.python.org/2/library/mmap.html

and an example: http://blog.schmichael.com/2011/05/15/sharing-python-data-between-processes-using-mmap/

• Geeocode: Thank you valentin though I already know mmap, but I thought it always writes to disk, but in one of the link I found a good counterexample, so I give you an up.