Interprocess messaging between two Python programs

Question

We have two Python programs running on two linux servers. Now we want to send messages between these Python programs. The best idea so far is to create a TCP/IP server and client architecture, but this seems like a very complicate way to do it. Is this really best practice for doing such a thing?

• jsalonen: What kind of messages you need to send between these programs? What are the roles of these servers: are the server running as peers or is there a master/slave kind of setting? What kind of latency you need to achieve? Also do you need to make sure all the messages arrive or is it OK if potentially some data is dropped on the way?
• Rohita Khatiwada: Its master slave setting. What I need to be done is, the master send message to all the slaves after it creates a table After receiving the message from the master, the slave will select rows from the master's table and do something. And now the slaves will send message to the master that it is done. SO here basically I need only the message for example something like , "done=True" to send to each other.
• jsalonen: So actually there are 2 or more server involved here? Is there also some specific database you are using? If so then why are you communicating between the servers with Python?

Answer 1

I like zeromq for simple messaging, it's really lightweight and fast...very flexible as well. Using AMQP messaging isn't a bad idea either depending on the specifics of your situation, I've found kombu to be a very nice pythonic library for that. You could also use xmlrpclib or setup a simple REST API with bottle or flask. Every option has it's place, so I'd investigate all your options.

• fouric: Would you happen to know if ZeroMQ works on Python 3? I examined the Python API for it and the code is Python 2.

Answer 2

This really depends on the kind of messaging you want and the roles of the two processes. If it's proper "client/server", I would probably create a SimpleHTTPServer and then use HTTP to communicate between the two. You can also use XMLRPCLib and the client to talk between them. Manually creating a TCP server with your own custom protocol sounds like a bad idea to me. You might also consider using a message queue system to communicate between them.

Answer 3

You could have a mulitprocessing.managers. As doc says :"A manager object controls a server process which manages shared objects. Other processes can access the shared objects by using proxies."

In your case, you could create a master process that control your other processes, each of those processes will call the master to grab the data.