python socket recv in multithread

Question

I have python script with only one socket object that is connect to a java server.

I started a thread for sending heart beat message to server per 5 secs.

And another thread for receiving message from server.

BTW, all the data send/recv is in protobuffer format.

# socket_client.py
def recv_handler():
 global client_socket
 while True:
 try:
 # read 4 bytes first 
 pack_len = client_socket.recv(4)
 pack_len = struct.unpack('!i', pack_len)[0]
 # read the rest
 recv_data = client_socket.recv(pack_len)
 # decode
 decompressed_data = data_util.decompressMessage(recv_data)
 sc_pb_message = data_util.decodePBMessage(decompressed_data)
 sc_head = data_util.parseHead(sc_pb_message)
 except:
 print 'error'
def heart_handler():
 global client_socket
 while True:
 if client_socket:
 message = data_util.makeMessage('MSG_HEART_BEAT')
 compressed_data = data_util.compressMessage(message)
 send_data = data_util.makeSendData(compressed_data)
 try: 
 client_socket.send(send_data)
 except: 
 print 'except'
 pass
 time.sleep(5)
def connect(address, port):
 global client_socket
 client_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
 client_socket.connect((address, port))
 # thread sending heart beat message
 th = threading.Thread(target = heart_handler)
 th.start()
 # thread recving message
 tr = threading.Thread(target = recv_handler)
 tr.start()

The code above works just fine. The script will send a heart beat message per 5 secs and receive the message from server and the message can be decoded successfully.

And here comes the trigger part than I do not know how to implement.

My python script need to receive input from the browser at the same time, so I started a BaseHTTPServer, to handle the POST request from the browser.

When a request come, I would like to call the client_socket.send method to send a specific message to the server and of course I need to return the data from server back to the browser.

# http_server.py
def do_POST(self):
 # ...
 result = socket_client.request(message)
 self.send_response(200)
 self.end_headers()
 self.wfile.write(...)

And here is what I tried to do in request:

def request(message):
 global client_socket
 client_socket.send(message)
 pack_len = client_socket.recv(4)
 pack_len = struct.unpack('!i', pack_len)[0]
 recv_data = client_socket.recv(pack_len)
 return recv_data

The problem I am having is the data I received in the request method after calling the send method seems to be disturbed by the data of heart beat in the thread.

If I comment out the heart beat thread and the receive thread, than the request method will work just fine. The data from server can decoded with no error and it can be sent back to the browser successfully.

My solution now might be wrong and I really do not know how to get this work.

Any advice will be appreciated, thanks :)

Answer 1

socket object in Python is not thread-safe, you need to access the shared resources (in this case the client_socket object) with the help of some synchronization primitives, such as threading.Lock in Python 2. Check here for a similar problem: Python: Socket and threads?