use Python urllib.urlopen to send data accurately

Question

I want to use Python to simulate a login action which acquires some message sending via HTTP GET method. So I write something like this

from urllib.request import urlopen, Request
urlopen(Request(URL, data=data_for_verify.encode(), method='GET'))

The problem is, it doesn't do the same as a real login action which like this (scratch from Wireshark, HTTP printable data only)

GET /rjsdcctrl?mac%3dfcaa14ec56f3%26ipv4%3d1681312010%26ipv61%3d0%26ipv62%3d0%26ipv63%3d0%26ipv64%3d0%26product%3d33554432%26mainver%3d67108864%26subver%3d1610612736 HTTP/1.1
Accept: text/*
User-Agent: HttpCall
Accept-Language: en-us
Host: 10.0.6.251
Cache-Control: no-cache

And what my program did is:

GET / HTTP/1.1
Accept-Encoding: identity
Content-Type: application/x-www-form-urlencoded
Host: 10.0.6.251:80
User-Agent: Python-urllib/3.4
Connection: close
Content-Length: 161
rjsdcctrl?mac%3dfcaa14ec56f3%26ipv4%3d1681312010%26ipv61%3d0%26ipv62%3d0%26ipv63%3d0%26ipv64%3d0%26product%3d33554432%26mainver%3d67108864%26subver%3d1610612736 

A real login action have the header comes first, and do not have the line GET / HTTP /1.1

or it just a header without content, and the first line GET contain the real request message. How can I simulate that using Python's urllib? I use Python 3.4

• Arne S: I would recomend the use of Requests (docs.python-requests.org) for tasks like this
• Amyth: As @Daniel states for a GET request you should add the parameters to the url. Using the data argument would actually send the data in the request body. Also to use headers you can use the keyword argument headers e.g. urlopen(url, headers=my_headers).

Answer 1

You shouldn't use the data parameter if you don't want to send data as part of the body. Append the value to the URL:

full_url = "%s?%s" % (URL, data_for_verify.encode())
urlopen(full_url)

• zl2003cn: I have tried your code. It comes like this GET /?b'rjsdcctrl?mac%3dfcaa14ec56f3%26ipv4%3d1681312010%26ipv61%3d0%26ipv62%3d0%26ipv63%3d0%26ipv64%3d0%26product%3d33554432%26mainver%3d67108864%26subver%3d1610612736 ' HTTP/1.1 It have a ?b at the beginning and surrounded by quote mark
• Daniel Roseman: I'm not sure what you are doing to get that. Are you passing the whole urlopen call somewhere? Please show your full code.

Answer 2

To extend @Daniel's answer, you can make use of urllib.urlencode method to prepare your get parameter string and also headers keyword argument to override default headers. So for example:

import urllib
url = 'http://www.example.com/'
data = {
 'key1': 'value1',
 'key2': 'value2',
 'key3': 'value3'
}
headers = {
 'Overriden-Header': 'Overriden Header Value'
}
## Update the url and make the actual requests
url = '%s?%s' % (url, urllib.urlencode(data))
response = urllib.urlopen(url, headers=headers)

• zl2003cn: Result comes like this GET /?b'rjsdcctrl?mac%3dfcaa14ec56f3%26ipv4%3d1681312010%26ipv61%3d0%26ipv62%3d0%26i‌​pv63%3d0%26ipv64%3d0%26product%3d33554432%26mainver%3d67108864%26subver%3d1610612‌​736 ' HTTP/1.1 It have a ?b at the beginning and surrounded by quote mark
• Amyth: That is because of your data_to_verify.encode() methods result. Make sure the data you pass to urlencode method is a python dictionary and is not already encoded. See how I have passed data to urllib.urlencode method in the example above.
• Amyth: Can you update your question with what data_for_verify.encode() method returns ?
• zl2003cn: Thank you! It is wrong using a raw string to encode, I use a dict instead and I got a proper response. By the way, urllib.urlencode method actually replace by urllib.parse.urlencode in Python 3.4 :)