Pass data to a subprocess

Wed Aug 1 06:59:37 EDT 2012

In article <mailman.2809.1343809166.4697.python-list at python.org>,
 Laszlo Nagy <gandalf at shopzeus.com> wrote:
> Yes, I think that is correct. Instead of detaching a child process, you 
> can create independent processes and use other frameworks for IPC. For 
> example, Pyro. It is not as effective as multiprocessing.Queue, but in 
> return, you will have the option to run your service across multiple 
> servers.

You might want to look at beanstalk (http://kr.github.com/beanstalkd/). 
We've been using it in production for the better part of two years. At 
a 30,000 foot level, it's an implementation of queues over named pipes 
over TCP, but it takes care of a zillion little details for you.
Setup is trivial, and there's clients for all sorts of languages. For a 
Python client, go with beanstalkc (pybeanstalk appears to be 
abandonware).
>> The most effective IPC is usually through shared memory. But there is no 
> OS independent standard Python module that can communicate over shared 
> memory.

It's true that shared memory is faster than serializing objects over a 
TCP connection. On the other hand, it's hard to imagine anything 
written in Python where you would notice the difference.