Connect to web server using python3 socket

Question

I'm running Python3 running Django on localhost and listening to http://127.0.0.1:8000/

and I want to test it by different ways like request and socket

import requests
res = requests.request("GET", "http://127.0.0.1:8000/")
txt = res.content
print(res)
res.close()
# <Response [200]>

It runs well, and

on my running Django console, it shows [07/Jul/2018 16:04:26] "GET / HTTP/1.1" 200 6744

Now instead requests to socket

import socket
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect(('127.0.0.1', 8000))
msg = b'GET / HTTP/1.1'
s.send(msg)
print(s)
s.close()
# <socket.socket fd=1324, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('127.0.0.1', 55312), raddr=('127.0.0.1', 8000)>

No error shows in my program

But I got the following exception on my running Django console

Invalid HTTP_HOST header: 'tony.jhou-1.xxservice.com:8000'. You may need to add 'tony.jhou-1.xxservice.com:8000' to ALLOWED_HOSTS.
[07/Jul/2018 16:08:10] "GET / HTTP/1.1" 400 63266
Traceback (most recent call last):
 File "C:\Users\tony.jhou\AppData\Local\Continuum\Anaconda3\lib\wsgiref\handlers.py", line 138, in run
 self.finish_response()
 File "C:\Users\tony.jhou\AppData\Local\Continuum\Anaconda3\lib\wsgiref\handlers.py", line 180, in finish_response
 self.write(data)
 File "C:\Users\tony.jhou\AppData\Local\Continuum\Anaconda3\lib\wsgiref\handlers.py", line 274, in write
 self.send_headers()
 File "C:\Users\tony.jhou\AppData\Local\Continuum\Anaconda3\lib\wsgiref\handlers.py", line 332, in send_headers
 self.send_preamble()
 File "C:\Users\tony.jhou\AppData\Local\Continuum\Anaconda3\lib\wsgiref\handlers.py", line 255, in send_preamble
 ('Date: %s\r\n' % format_date_time(time.time())).encode('iso-8859-1')
 File "C:\Users\tony.jhou\AppData\Local\Continuum\Anaconda3\lib\wsgiref\handlers.py", line 453, in _write
 result = self.stdout.write(data)
 File "C:\Users\tony.jhou\AppData\Local\Continuum\Anaconda3\lib\socketserver.py", line 775, in write
 self._sock.sendall(b)
 File "D:\emc\ve\lib\site-packages\gevent\_socket3.py", line 459, in sendall
 return _socketcommon._sendall(self, data_memory, flags)
 File "D:\emc\ve\lib\site-packages\gevent\_socketcommon.py", line 358, in _sendall
 timeleft = __send_chunk(socket, chunk, flags, timeleft, end)
 File "D:\emc\ve\lib\site-packages\gevent\_socketcommon.py", line 287, in __send_chunk
 data_sent += socket.send(chunk, flags)
 File "D:\emc\ve\lib\site-packages\gevent\_socket3.py", line 440, in send
 return _socket.socket.send(self._sock, data, flags)
ConnectionAbortedError: [WinError 10053] 連線已被您主機上的軟體中止。
[07/Jul/2018 16:08:10] "GET / HTTP/1.1" 500 59
----------------------------------------
Exception happened during processing of request from ('127.0.0.1', 54587)
Traceback (most recent call last):
 File "C:\Users\tony.jhou\AppData\Local\Continuum\Anaconda3\lib\wsgiref\handlers.py", line 138, in run
 self.finish_response()
 File "C:\Users\tony.jhou\AppData\Local\Continuum\Anaconda3\lib\wsgiref\handlers.py", line 180, in finish_response
 self.write(data)
 File "C:\Users\tony.jhou\AppData\Local\Continuum\Anaconda3\lib\wsgiref\handlers.py", line 274, in write
 self.send_headers()
 File "C:\Users\tony.jhou\AppData\Local\Continuum\Anaconda3\lib\wsgiref\handlers.py", line 332, in send_headers
 self.send_preamble()
 File "C:\Users\tony.jhou\AppData\Local\Continuum\Anaconda3\lib\wsgiref\handlers.py", line 255, in send_preamble
 ('Date: %s\r\n' % format_date_time(time.time())).encode('iso-8859-1')
 File "C:\Users\tony.jhou\AppData\Local\Continuum\Anaconda3\lib\wsgiref\handlers.py", line 453, in _write
 result = self.stdout.write(data)
 File "C:\Users\tony.jhou\AppData\Local\Continuum\Anaconda3\lib\socketserver.py", line 775, in write
 self._sock.sendall(b)
 File "D:\emc\ve\lib\site-packages\gevent\_socket3.py", line 459, in sendall
 return _socketcommon._sendall(self, data_memory, flags)
 File "D:\emc\ve\lib\site-packages\gevent\_socketcommon.py", line 358, in _sendall
 timeleft = __send_chunk(socket, chunk, flags, timeleft, end)
 File "D:\emc\ve\lib\site-packages\gevent\_socketcommon.py", line 287, in __send_chunk
 data_sent += socket.send(chunk, flags)
 File "D:\emc\ve\lib\site-packages\gevent\_socket3.py", line 440, in send
 return _socket.socket.send(self._sock, data, flags)
ConnectionAbortedError: [WinError 10053] 連線已被您主機上的軟體中止。
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
 File "C:\Users\tony.jhou\AppData\Local\Continuum\Anaconda3\lib\wsgiref\handlers.py", line 141, in run
 self.handle_error()
 File "D:\emc\ve\lib\site-packages\django\core\servers\basehttp.py", line 88, in handle_error
 super(ServerHandler, self).handle_error()
 File "C:\Users\tony.jhou\AppData\Local\Continuum\Anaconda3\lib\wsgiref\handlers.py", line 368, in handle_error
 self.finish_response()
 File "C:\Users\tony.jhou\AppData\Local\Continuum\Anaconda3\lib\wsgiref\handlers.py", line 180, in finish_response
 self.write(data)
 File "C:\Users\tony.jhou\AppData\Local\Continuum\Anaconda3\lib\wsgiref\handlers.py", line 274, in write
 self.send_headers()
 File "C:\Users\tony.jhou\AppData\Local\Continuum\Anaconda3\lib\wsgiref\handlers.py", line 331, in send_headers
 if not self.origin_server or self.client_is_modern():
 File "C:\Users\tony.jhou\AppData\Local\Continuum\Anaconda3\lib\wsgiref\handlers.py", line 344, in client_is_modern
 return self.environ['SERVER_PROTOCOL'].upper() != 'HTTP/0.9'
TypeError: 'NoneType' object is not subscriptable
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
 File "C:\Users\tony.jhou\AppData\Local\Continuum\Anaconda3\lib\socketserver.py", line 639, in process_request_thread
 self.finish_request(request, client_address)
 File "C:\Users\tony.jhou\AppData\Local\Continuum\Anaconda3\lib\socketserver.py", line 361, in finish_request
 self.RequestHandlerClass(request, client_address, self)
 File "C:\Users\tony.jhou\AppData\Local\Continuum\Anaconda3\lib\socketserver.py", line 696, in __init__
 self.handle()
 File "D:\emc\ve\lib\site-packages\django\core\servers\basehttp.py", line 155, in handle
 handler.run(self.server.get_app())
 File "C:\Users\tony.jhou\AppData\Local\Continuum\Anaconda3\lib\wsgiref\handlers.py", line 144, in run
 self.close()
 File "C:\Users\tony.jhou\AppData\Local\Continuum\Anaconda3\lib\wsgiref\simple_server.py", line 35, in close
 self.status.split(' ',1)[0], self.bytes_sent
AttributeError: 'NoneType' object has no attribute 'split'

I'm new to socket programming and just start learning networking, I think maybe I should send more request head. Please tell me how to do that, thanks.

Answer 1

Yes, you need to send more data. A minimal HTTP 1.1 request requires a second line that contains a Host header specifying the name or address of the host that should handle the request, and the request header lines must be followed by an empty line to tell the server that the header has been completely received.

In HTTP the end of a line is indicated by a pair of characters, a carriage return and a line feed. In Python strings \r represents a carriage return and \n represents a line feed. (The line feed character is sometimes called a newline character, hence the \n code.) This means that you have to write \r\n to mark the end of each line in the request.

So, what you need is:

msg = b'GET / HTTP/1.1\r\nHost: 127.0.0.1\r\n\r\n'

The final \r\n produces the empty line that indicates the end of the headers.

After you make that change it's likely that your server will report a different error, because your client program closes its socket immediately. That breaks the connection between the client and the server. When the server tries to send its response, it will find that the connection has been broken and it will probably complain about that. To avoid that, change your client to use s.recv to wait for and read the response from the server after it does s.send(). Something like:

while True:
 resp = s.recv(8000)
 if resp:
 print resp
 else:
 break

• Tony Chou: You save my day :-)