Sending data from outside socket object in python

Question

I made a Client socket object, which I instantiate and it keeps alive a connection with the server, which is working fine, but I'm wondering if there is a way to call the socket.send event from outside the instance. I was about to make a stack for the messages and check the stack in the while loop and if it's not empty then send the oldest data to the server, which would be just fine for me, but my problem is that the stack only updates after the while loop(I tried breaking out, then it updated). So my question would be, is there a way to update the global stack simultaneously with the while loop running? Or is there any other way to call the socket.send event outside the object?

import socket
import sys
import select
import threading
SERVER_IP = '192.168.1.4'
PORT = 8686
TIMEOUT = 5
BUF_SIZE = 1024
MESSAGES = ['testdata1', 'testdata2']
class Client(threading.Thread):
 def __init__(self, host=SERVER_IP, port=PORT):
 threading.Thread.__init__(self)
 self.sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
 self.sock = socket.create_connection((host, port), 1)
 self.sock.setblocking(0)
 while 1:
 try:
 global MESSAGES
 ready = select.select([self.sock], [], [], TIMEOUT*1000)
 if ready[0]:
 buf = self.sock.recv(BUF_SIZE)
 print buf
 #TODO:do stuff with buf
 print 'messages left:'+str(len(MESSAGES))
 if len(MESSAGES)>0:
 self.sock.send(MESSAGES.pop())
 except KeyboardInterrupt:
 self.sock.close()
 sys.exit(1)
 except Exception, e:
 print '\n[ERR] %s' % e
 self.sock.close()
 sys.exit(1)
 def run(self):
 pass
 def sendData(self, data):
 global MESSAGES
 print 'appending data:%s' % data
 MESSAGES.append(data)
def main():
 client = Client()
 client.start()
 client.sendData("test1")
 client.sendData("test2")
 client.sendData("test3")
if __name__ == '__main__':
 try:
 main()
 except KeyboardInterrupt:
 sys.exit(1)

Answer 1

Client.__init__() does not return because it enters an infinite while loop. Hence control is never returned to the main thread, and the Client thread is not actually started.

Instead you should move the while loop into the run() method. Then the __init__() method will return control to the main thread, which can then start the thread, and request that the client send messages via sendData().

class Client(threading.Thread):
 def __init__(self, host=SERVER_IP, port=PORT):
 threading.Thread.__init__(self)
 self.sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
 self.sock = socket.create_connection((host, port), 1)
 self.sock.setblocking(0)
 def run(self):
 while 1:
 try:
 global MESSAGES
 ready = select.select([self.sock], [], [], TIMEOUT*1000)
 if ready[0]:
 buf = self.sock.recv(BUF_SIZE)
 print buf
 #TODO:do stuff with buf
 print 'messages left:'+str(len(MESSAGES))
 if len(MESSAGES)>0:
 self.sock.send(MESSAGES.pop())
 except KeyboardInterrupt:
 self.sock.close()
 sys.exit(1)
 except Exception, e:
 print '\n[ERR] %s' % e
 self.sock.close()
 sys.exit(1)
 def sendData(self, data):
 global MESSAGES
 print 'appending data:%s' % data
 MESSAGES.append(data)

Instead of using the global MESSAGES list you should probably create a Queue for communicating between the main thread and the worker thread(s), particularly if more than one worker thread is running. Something like this (untested!):

import Queue
class Client(threading.Thread):
 def __init__(self, msg_queue, host=SERVER_IP, port=PORT):
 threading.Thread.__init__(self)
 self.msg_queue = msg_queue
 self.sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
 self.sock = socket.create_connection((host, port), 1)
 self.sock.setblocking(0)
 def run(self):
 while 1:
 try:
 ready = select.select([self.sock], [], [], TIMEOUT*1000)
 if ready[0]:
 buf = self.sock.recv(BUF_SIZE)
 print buf
 #TODO:do stuff with buf
 print 'messages left:'+ str(self.msg_queue.qsize())
 try:
 msg = self.msg_queue.get_nowait()
 self.sock.send(msg)
 except Queue.Empty:
 pass
 except KeyboardInterrupt:
 self.sock.close()
 sys.exit(1)
 except Exception, e:
 print '\n[ERR] %s' % e
 self.sock.close()
 sys.exit(1)
def main():
 # create a queue and pass it to the client
 msg_queue = Queue.Queue()
 client = Client(msg_queue)
 client.start()
 msg_queue.put("test1")
 msg_queue.put("test2")
 msg_queue.put("test3")

• Dalen: As your answer contains the same info as mine, but you added some code, please consider adding code without select as well, as I proposed. Or I will add it to my answer. Or else using Queue() for speed doesn't make too much sense for speeding up the thing. I think this is a small client app, so difference between these data types wouldn't matter much. But bothering OS with unnecessary system calls will increase CPU usage.
• mhawke: @Dalen : well, my answer contains some of the same info as yours, but mine does go further to suggest a thread safe way of communicating between the main thread and (potentially) multiple worker threads. This is a better architecture going forward, and certainly better than using a global variable. Regarding speed, what is the issue? The OP didn't mention one, and Queue is not slow.
• mhawke: @Dalen: regarding the use of select() within the thread is not, as you say, ideal. In this example it would be better to simply perform a blocking recv(), although it is not difficult to imagine that the main thread might want to send messages to the worker thread to control it, e.g. a quit command. In that case use of select() is not an unreasonable, and for that reason I will leave my answer as it is.
• Dalen: I agree, but as I explained, it is not an issue of the client's speed. It just makes one system call more than necessary. select() is fast, yes, but why bothering OS first with syscall, then select() itself checks the socket over and over for 5 seconds. If TIMEOUT is smaller, select() bothers socket less, but we bother OS more. As for thread safe, well, first, I'd advice the OP not to store messages in a global variable if possible. And second, there is only one listener registered, and no reuse_addr(), and global msgs. That's why I assumed that there will be always only one Client() per app.
• Dalen: Yes, this concept will work, if you want to send data, but if nonblocking mode is used, recv() is, well, nonblocking. That means that timeout can be much smaller than 5 seconds, so we can send data at any time, and receive data faster. And OS is free but for a socket. I think that OS performs something similar to select() anyway, when socket is in non blocking mode. But as it is all in C and direct on our socket it is better for all. Well I think that that's enough on this subject. I feel that I am boring you all. :D Just good intentions for further improvements.

Answer 2

The thing should work if you move your loop from

__init__() into run()

method instead.

Your thread is not a thread this way, process blocks at client = Client(...).

Why do you mix select and threads? Is this really necessary? If you want asynchronous sending and receiving without threads use asyncore module. Or remove select from your code. The socket.recv() will block until it receives data in blocking mode, but as this is a thread, I don't see anything wrong about that. If in nonblocking mode, recv() will just return None if there is no data to receive if I remember correctly. So you don't really need select. Just check if recv() returned None. If it does, sleep some time before trying again.

The way you did it troubles your OS twice. Once for reading a socket, and second time to get the status of a socket where timeout is used to simulate sleep() more than anything else. Then the loop checks again making select() system call right after timeout confirmed that there is nothing to do for that socket.