Send and receive objects through sockets in Python

Question

I have searched a lot on the Internet, but I haven't been able to find the solution to send an object over the socket and receive it as is. I know it needs pickling which I have already done. And that converts it to bytes and is received on the other hand. But how can I convert those bytes to that type of object?

process_time_data = (current_process_start_time, current_process_end_time)
prepared_process_data = self.prepare_data_to_send(process_time_data)
data_string = io.StringIO(prepared_process_data)
data_string = pack('>I', len(data_string)) + data_string
self.send_to_server(data_string)

This is the code which is converting the object to StringIO on the client and sending to the server. And on the server side I am getting bytes. Now I am searching for bytes to be converted to StringIO again so that I can get the object value.

In the code, Object is wrapped in StringIO and is being sent over the socket. Is there a better approach?

The server-side code is as follows.

server = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
server.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
#server.setblocking(0)
server.bind(('127.0.0.1', 50000))
server.listen(5)
inputs = [server]
outputs = []
message_queues = {}
while inputs:
 readable, writeable, exceptional = select.select(inputs, outputs, inputs)
 for s in readable:
 if s is server:
 connection, client_address = s.accept()
 print(client_address)
 connection.setblocking(0)
 inputs.append(connection)
 message_queues[connection] = queue.Queue()
 print('server started...')
 else:
 print('Getting data step 1')
 raw_msglen = s.recv(4)
 msglen = unpack('>I', raw_msglen)[0]
 final_data = b''
 while len(final_data) < msglen:
 data = s.recv(msglen - len(final_data))
 if data:
 #print(data)
 final_data += data
 message_queues[s].put(data)
 if s not in outputs:
 outputs.append(s)
 else:
 if s in outputs:
 outputs.remove(s)
 else:
 break
 inputs.remove(connection)
 #s.close()
 del message_queues[s]
 process_data = ProcessData()
 process_screen = ProcessScreen()
 if final_data is not None:
 try:
 deserialized_data = final_data.decode("utf-8")
 print(deserialized_data)
 except (EOFError):
 break
 else:
 print('final data is empty.')
 print(process_data.project_id)
 print(process_data.start_time)
 print(process_data.end_time)
 print(process_data.process_id)

The two helper functions are as follows:

def receive_all(server, message_length, message_queues, inputs, outputs):
 # Helper function to recv message_length bytes or return None if EOF is hit
 data = b''
 while len(data) < message_length:
 packet = server.recv(message_length - len(data))
 if not packet:
 return None
 data += packet
 message_queues[server].put(data)
 if server not in outputs:
 outputs.append(server)
 else:
 if server in outputs:
 outputs.remove(server)
 inputs.remove(server)
 del message_queues[server]
 return data
def receive_message(server, message_queues, inputs, outputs):
 # Read message length and unpack it into an integer
 raw_msglen = receive_all(server, 4, message_queues, inputs, outputs)
 if not raw_msglen:
 return None
 message_length = unpack('>I', raw_msglen)[0]
 return receive_all(server, message_length, message_queues, inputs, outputs)

And two of the model classes are as follows:

class ProcessData:
 process_id = 0
 project_id = 0
 task_id = 0
 start_time = 0
 end_time = 0
 user_id = 0
 weekend_id = 0
# Model class to send image data to the server
class ProcessScreen:
 process_id = 0
 image_data = bytearray()

• Sudheesh Singanamalla: Please take some time to read the help page, especially the sections named "What topics can I ask about here?" and "What types of questions should I avoid asking?". And more importantly, please read the Stack Overflow question checklist. You might also want to learn about Minimal, Complete, and Verifiable Examples.
• user955340: @Dharmindar "once my teacher said no question is bad" - they are wrong, there are definitely bad questions. This is one of them. See suggested duplicate below for what information is needed when asking a question, and also for the answer that will help you.
• Erick Shepherd: As @SudheeshSinganamalla said, your example of the problem should be minimal, complete, and verifiable. This is not a complete example; it is not a standalone code block: that is, it cannot be run as is. Without the respective definitions, the only context we have for the variables and functions being used is their names. This makes it very difficult to answer the question at all, let alone adequately, which is why it's being downvoted: The purpose of the vote is a kind of triage to rank questions according to priority. This may or may not be a bad question, but it's certainly poorly asked.
• Erick Shepherd: @Dharmindar You need to give us more to work with. You can edit the question to address the concerns I and other users commenting here have expressed: Chiefly, include the rest of the example code (complete), minus any superfluous blocks that are not relevant to the question (minimal) with a single repeatable issue that we can address (verifiable). Otherwise, you are not likely to get an answer.
• Sudheesh Singanamalla: You seem to have made comments where you've asked me to stay away. If you had edited to question to look remotely as it looks now when I made my comment, you'd have received an answer in the meanwhile. The reason why we moderate is so that someone experienced can answer your question but when your question is not clear or takes up way too much time, no one would spend time to answer it. Good luck!

Answer 1

You're looking for pickle and the loads and dumps operations. Sockets are basically byte streams. Let us consider the case you have.

class ProcessData:
 process_id = 0
 project_id = 0
 task_id = 0
 start_time = 0
 end_time = 0
 user_id = 0
 weekend_id = 0

An instance of this class needs to be pickled into a data string by doing data_string = pickle.dumps(ProcessData()) and unpickled by doing data_variable = pickle.loads(data) where data is what is received.

So let us consider a case where the client creates an object of ProcessData and sends it to server. Here's what the client would look like. Here's a minimal example.

Client

import socket, pickle
class ProcessData:
 process_id = 0
 project_id = 0
 task_id = 0
 start_time = 0
 end_time = 0
 user_id = 0
 weekend_id = 0
HOST = 'localhost'
PORT = 50007
# Create a socket connection.
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect((HOST, PORT))
# Create an instance of ProcessData() to send to server.
variable = ProcessData()
# Pickle the object and send it to the server
data_string = pickle.dumps(variable)
s.send(data_string)
s.close()
print 'Data Sent to Server'

Now your server which receives this data looks as follows.

Server

import socket, pickle
class ProcessData:
 process_id = 0
 project_id = 0
 task_id = 0
 start_time = 0
 end_time = 0
 user_id = 0
 weekend_id = 0
HOST = 'localhost'
PORT = 50007
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.bind((HOST, PORT))
s.listen(1)
conn, addr = s.accept()
print 'Connected by', addr
data = conn.recv(4096)
data_variable = pickle.loads(data)
conn.close()
print data_variable
# Access the information by doing data_variable.process_id or data_variable.task_id etc..,
print 'Data received from client'

Running the server first creates a bind on the port and then running the client makes the data transfer via the socket. You could also look at this answer.

• Sudheesh Singanamalla: What do you mean you've run it and it doesn't work. How are you running the individual files? Can you open two terminals, copy the code above into the respective files server.py and client.py. Run the server in one terminal and client in another. You'll get a <__main__.ProcessData instance at 0x10572edd0> as a response which is an instance of ProcessData class you've sent from client to server.
• user955340: @Dharmindar This example works perfectly, shows and explains what you need to do PERFECTLY. This solves the question you asked, please mark it as accepted.
• Sudheesh Singanamalla: @Bilkokuya Thank you very much. @Dharmindar Glad it helped. However, I would like to tell you something. You mentioned in your comments above Instead responding to my last comment you could have answered the question or could have said that it is out of my knowledge. Please understand we are not doing you a favour and you shouldn't be speaking to anyone in the stackoverflow community this way. All the best with your work.
• Dharmindar: @SudheeshSinganamalla will keep that in mind. Thanks again :)
• Sudheesh Singanamalla: The issue is because of the limit 4096 bytes. You need a loop to receive all the data and then join the 4096 byte chunks into the string you want. Here is an example. You can easily search for these on stackoverflow. Please don't expect us to write everything.

Answer 2

Shameless plug here, but a friend and I have recently released tlspyo, an open-source library whose purpose is to help you transfer python objects over network easily and in a secure fashion.

Transferring pickled objects via Internet sockets without using something like tlspyo is basically an open door for hackers, so don't do it.

With tlspyo, your code looks like this:

Server:

from tlspyo import Relay
if __name__ == "__main__":
 my_server = Relay(port=3000,password="<same strong password>")
 # (...)

Client 1:

from tlspyo import Endpoint
if __name__ == "__main__":
 client_1 = Endpoint(
 ip_server='<ip of your server>'
 port=3000,
 password="<same strong password>",
 groups="client 1")
 # send an object to client 2:
 my_object = "my object" # doesn't have to be a string, of course
 client_1.broadcast(my_object, "client 2")
 # (...)

Client 2:

from tlspyo import Endpoint
if __name__ == "__main__":
 client_2 = Endpoint(
 ip_server='<ip of my Relay>'
 port=3000,
 password="<same strong password>",
 groups="client 2")
 # receive the object sent by client 1:
 my_object = client_2.receive_all(blocking=True)[0]
 # (...)

(You will need to setup TLS for this code to work, check out the documentation - or you can disable TLS using security=None, but if you are transferring over the Internet you don't want to do that.)

Answer 3

An option is to use JSON serialization.

However, Python objects are not serializable, so you have to map your class object into Dict first, using either function vars (preferred) or the built-in __dict__.

Adapting the answer from Sudheesh Singanamalla and based on this answer:

Client

import socket, json
class ProcessData:
 process_id = 0
 project_id = 0
 task_id = 0
 start_time = 0
 end_time = 0
 user_id = 0
 weekend_id = 0
HOST = 'localhost'
PORT = 50007
# Create a socket connection.
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect((HOST, PORT))
# Create an instance of ProcessData() to send to server.
variable = ProcessData()
# Map your object into dict
data_as_dict = vars(variable)
# Serialize your dict object
data_string = json.dumps(data_as_dict)
# Send this encoded object
s.send(data_string.encode(encoding="utf-8"))
s.close()
print 'Data Sent to Server'

Server

import socket, json
class ProcessData:
 process_id = 0
 project_id = 0
 task_id = 0
 start_time = 0
 end_time = 0
 user_id = 0
 weekend_id = 0
HOST = 'localhost'
PORT = 50007
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.bind((HOST, PORT))
s.listen(1)
conn, addr = s.accept()
print 'Connected by', addr
data_encoded = conn.recv(4096)
data_string = data_encoded.decode(encoding="utf-8")
data_variable = json.loads(data_string)
# data_variable is a dict representing your sent object
conn.close()
print 'Data received from client'

Warning

One important point is that dict mapping of an object instance does not map class variable, only instance variable. See this answer for more information. Example:

class ProcessData:
 # class variables
 process_id = 0
 project_id = 1
 def __init__(self):
 # instance variables
 self.task_id = 2
 self.start_time = 3
obj = ProcessData()
dict_obj = vars(obj)
print(dict_obj)
# outputs: {'task_id': 2, 'start_time': 3}
# To access class variables:
dict_class_variables = vars(ProcessData)
print(dict_class_variables['process_id'])
# outputs: 0