Memory management with Python multiprocessing

Question

The way I understand it, another Python instance is kicked off with multiprocessing. If so,

a. Is a Python instance started for each multiprocessing process?

b. If one process is working on say, an in-memory database table and another process is working on another in-memory database table, how does Python manage the memory allocation for the 2 processes?

c. Wrt b), is the memory allocation persistant between invocations ie. if the first process is used continuously but the 2nd process is used infrequently, is the in-memory table re-created between process calls to it?

• User: 1. What do you mean by memory table? 2. Is your question only specific to a certain operating system or targeting the inside of a python process?
• Henry Thornton: the data structure is an "in-memory database table", could be a list or dictionary. question is not specific to a particular operating system.

Answer 1

(a) yes

(b) Python pretty much doesn't manage it, the OS does

(c) Yes, if the second process exits then its resources are freed, regardless of the persistence of the first process. You can in principle use shared objects to allow the second process to use something that the first process arranges will persist. How that plays with the specific example of the "something" being a database table is another matter.

Running extra Python processes with multiprocessing is a lot like running extra Python (or for that matter Java) processes with subprocess. The difference is that multiprocessing gives you a suite of ways to communicate between the processes. It doesn't change the basic lifecycle and resource-handling of processes by the OS.

• Henry Thornton: Ah! If we make it simpler and use a list (or dictionary) as the data structures being processed, then: The main Python process (#1) initializes list #1 and list #2. Define list #2 as a shared object. Process #1 spawns process #2 to work on list #2, and returns result to process #1 and dies. Next time process #1 needs to work on list #2, it spawns (a new) process #2. Is that about it?
• Steve Jessop: @dbv: yes, that should work, using multiprocessing.Array as the type of the shared object. Or if process #2 doesn't modify list #2, just computes a result from it and returns that, then you don't need shared memory at all on a proper operating system -- you can fork instead and the OS will magically arrange that memory neither process modifies is never copied. Unfortunately Windows is not a proper OS. I define "proper" to mean "provides fork" ;-)
• Henry Thornton: terrific! process #2 does modify list #2 and so shared memory is the way to go. a related question: the concept behind Python multiprocessing is to take advantage of multi-core cpu's ie. process #2 would be executed by a different core-cpu than process #1. what would happen if the machine was a one-core cpu?
• Steve Jessop: Multi-processing (and multi-threading) on a one-core CPU works the same ways a multi-core CPUs work when there are more processes (or threads) ready to run than there are cores to run them on. Something runs for a while and then is interrupted and suspended to let something else run. The OS chooses what to run according to priority and whatnot.