fast-ish python/jython IPC?

Question

All I want to do is make some RPC calls over sockets. I have a server that does backendish stuff running jython 2.5. I need to make some calls from a frontend server running Django on CPython. I've been beating my head against a wall getting any form of IPC going.

The list of things I've tried:

• Apache Thrift doesn't have any actual releases, just snapshots. I'd like to use something stable.
• JSON-RPC is interesting, and it should be able to run over sockets, but in practice most of the implementations only seem to work over HTTP. HTTP overhead is exactly what I'm trying to avoid.
• Protocol Buffers is really only a serialization protocol. From what I gather protobuf provides interface generation for RPC, but it's only the interface. Actually writing all the connection code is up to the user. If I'm going to be stuck using sockets, I'll just use JSON for serialization. It's simpler and faster.
• Pyro doesn't work properly with Jython as a server. Some sort of socket timeout issue. I've sent a message to the mailing list.
• pysage Yay for message passing! Only it requires python 2.6 or the processing module (which has compiled extensions). Jython is version 2.5 and doesn't allow compiled extensions.
• Candygram is an interesting alternative to pysage, but as far as I can tell it's unmaintained. I haven't even tried it out with Jython. Any experiences with it?
• Twisted Perspective Broker Twisted doesn't work on Jython.

I know that it'd be a snap doing this with XML-RPC, which makes me even more cranky. I want to avoid the overhead of HTTP, but at the same time I really don't want to get down and dirty with sockets to implement my own protocol. I'll do it wrong if I do.

Any ideas? I'm probably going to cry for about 20 minutes and then just use XML-RPC.

• Kobold: btw, I just found this solution to my question: bert-rpc.org

Answer 1

Have you considered Hessian? From the blurb:

The Hessian binary web service protocol makes web services usable without requiring a large framework, and without learning yet another alphabet soup of protocols. Because it is a binary protocol, it is well-suited to sending binary data without any need to extend the protocol with attachments.

It has Python client and Java server (and more besides).

Update: If you're dead against HTTP, why not just use SocketServer and pickle? Not much of a protocol needed, hard to get wrong. Send / receive pickled strings with length prefixes.

• Kobold: Hessian uses HTTP, which I'd like to avoid.

Answer 2

How about using sockets, but with the help of asyncore and asynchat?

Some links:

• An example
• http://docs.python.org/library/asyncore.html
• http://docs.python.org/library/asynchat.html

• pts: Some other asynchronous frameworks: Twisted, gevent, Eventlet, Syncless, Concurrence.

Answer 3

Two that look the most interesting to me:

• Gearman and Python bindings. It's quite a bit faster now that it's been re-written in C (originally perl). It's used in production (although I can't point to any examples of the python bindings being used in production). It has very interesting (to me) interfaces into MySQL and Postgresql. Finally, todays tweet from Django's Jacob Kaplan-Moss.

• RabbitMQ although because it's just a message queue you'll still have to serialize your own messages unless you also use celery.

• Kobold: Currently we're using carrot/RabbitMQ to make calls to this server asynchronously, but now we'd like to make synchronous calls. I'm sure I could build that over the queue, but it's not obvious how to do that in a good way.

Answer 4

My favorite.. zeroc's ice

• Kobold: Ice does look very interesting. We're not an open project, so I'll look into how much it costs for a commercial license.

Answer 5

Have you thought about using CORBA? It's fast, portable and object oriented...

I have only used it on the Java side (I think you could use a pure java broker with no problems from Jython), and with IIOP you should be able to interoperate with the CPython client.

• Kobold: It may very well come to that. Right now our interface is evolving pretty fast, and CORBA seems too heavyweight. But yes, the thought has occurred to me.
• fortran: I don't think its too heavyweight, in the sense of performance or memory footprint, it's used in embedded systems too... if you mean it can be too cumbersome to deploy all the services it can provide and you won't probably use, yes, then you're very right ^_^