Pickling over a socket

Tue Apr 19 15:30:53 EDT 2011

On Tue, Apr 19, 2011 at 11:53 AM, Roger Alexander <rtalexander at mac.com> wrote:
> Hi,
>> I'm trying to understand how to pickle Python objects over a TCP
> socket.
>> In the example below (based on code from Foundations of Python Network
> Programming), a client creates a dictionary (lines 34-38) and uses
> pickle.dump at line 42 to write the pickled object using file handle
> make from a socket. The server de-pickles with pickle.load  (line 24),
> again using a file handle made from a socket.
>> When I run the program, the following output is produced:
>>    Listening at ('127.0.0.1', 1060)
>    Accepted connection from ('127.0.0.1', 49938)
>    Traceback (most recent call last):
>    File "pickles.py", line 24, in <module>
>        d = pickle.load( s_fh )
>    File "/usr/local/lib/python2.7/pickle.py", line 1378, in load
>        return Unpickler(file).load()
>    File "/usr/local/lib/python2.7/pickle.py", line 857, in load
>        key = read(1)
>    File "/usr/local/lib/python2.7/socket.py", line 380, in read
>        data = self._sock.recv(left)
>    socket.error: [Errno 107] Transport endpoint is not connected
>> I'm at a loss, can anyone provide any guidance?
>> Thanks,
>> Roger Alexander

I played around with it until something worked, and ended up with the
below. The most significant change was probably using sc.makefile
instead of s.makefile in the server, but I seemed to need some data
framing too despite the pickling. It's possible you won't need that
if you just flush your file in the client; I don't much pickling
experience - in particular, I don't know if you can concatenate
pickled objects and load them serially from a file-like object without
any (additional) framing.
I like to use bufsock for this sort of stuff, but I'm probably unique
in that. ^_^ http://stromberg.dnsalias.org/~strombrg/bufsock.html
#!/usr/bin/python
import time
import pickle
import socket, sys
import pprint
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
HOST = sys.argv.pop() if len(sys.argv) == 3 else '127.0.0.1'
PORT = 1060
if sys.argv[1:] == ['server']:
 s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
 s.bind((HOST, PORT))
 s.listen(1)
 print 'Listening at', s.getsockname()
 sc, sockname = s.accept()
 print 'Accepted connection from', sockname
 sf = sc.makefile( "rb" )
 length_list = []
 while True:
 char = sf.read(1)
 if char == '\n':
 break
 else:
 length_list.append(int(char))
 length = 0
 for digit in length_list:
 length = length * 10 + digit
 data = sf.read(length)
 d = pickle.loads(data)
 pprint.pprint(d)
 sc.shutdown(socket.SHUT_RDWR)
 sc.close()
 s.close()
elif sys.argv[1:] == ['client']:
 s.connect((HOST, PORT))
 # s.shutdown(socket.SHUT_RD)
 d = dict()
 d[ 'Name' ] = 'Jake Thompson.'
 d[ 'Age' ] = 25
 d[ 'Location' ] = 'Washington, D.C.'
 sf = s.makefile( "wb" )
 string = pickle.dumps( d, pickle.HIGHEST_PROTOCOL )
 sf.write('%d\n' % len(string))
 sf.write(string)
 sf.flush()
 #time.sleep(10)
 sf.close()
 s.shutdown(socket.SHUT_RDWR)
 # s.close()
else:
 print >>sys.stderr, 'usage: streamer.py server|client [host]'