homepage

httplib module's auxiliary class LineAndFileWrapper contains a readline() method having no arguments (than self). It's being used in code where commented with "assume it's a Simple-Response from an 0.9 server" as a wrapper for response stream ("self.fp = LineAndFileWrapper(line, self.fp)").
In some cases readline() method requires a size argument (e.g. used with size argument inside HTTPMessage/readheaders(), HTTPResponse/begin(), HTTPResponse/_read_chunked, HTTPConnection/_tunnel, at least in "Python 2.7.1+ (r271:86832, Sep 27 2012, 21:12:17)").
This bug is hard to be reproduced as it requires a little help of "divine intervention".
One such exception looks like (reproduced on one user machine):
...
conn = urllib2.urlopen(req)
File "/usr/lib/python2.7/urllib2.py", line 127, in urlopen
return _opener.open(url, data, timeout)
File "/usr/lib/python2.7/urllib2.py", line 401, in open
response = self._open(req, data)
File "/usr/lib/python2.7/urllib2.py", line 419, in _open
'_open', req)
File "/usr/lib/python2.7/urllib2.py", line 379, in _call_chain
result = func(*args)
File "/usr/share/sqlmap/lib/request/httpshandler.py", line 64, in https_open
return self.do_open(HTTPSConnection if ssl else httplib.HTTPSConnection, req)
File "/usr/lib/python2.7/urllib2.py", line 1178, in do_open
h.request(req.get_method(), req.get_selector(), req.data, headers)
File "/usr/lib/python2.7/httplib.py", line 962, in request
self._send_request(method, url, body, headers)
File "/usr/lib/python2.7/httplib.py", line 996, in _send_request
self.endheaders(body)
File "/usr/lib/python2.7/httplib.py", line 958, in endheaders
self._send_output(message_body)
File "/usr/lib/python2.7/httplib.py", line 818, in _send_output
self.send(msg)
File "/usr/lib/python2.7/httplib.py", line 780, in send
self.connect()
File "/usr/share/sqlmap/lib/request/httpshandler.py", line 46, in connect
sock = create_sock()
File "/usr/share/sqlmap/lib/request/httpshandler.py", line 39, in create_sock
self._tunnel()
File "/usr/lib/python2.7/httplib.py", line 748, in _tunnel
line = response.fp.readline(_MAXLINE + 1)
TypeError: readline() takes exactly 1 argument (2 given)

Mentioned "divine intervention" could be:
1) Argument strict should be set to 0/False when instantiating HTTPResponse
2) No status should be returned from the backend server (hence the "# assume it's a Simple-Response from an 0.9 server")
3) Any part of response should be read (body, headers, etc.) as HTTPResponse's method begin() uses "self.fp.readline(_MAXLINE + 1)"

Thanks for the report. Looks like it is a fairly rare case, which I suppose is why it hasn't been reported before.
If the "could be" turns out to be sufficient, that should be enough for someone to write a test for this. The patch itself should be even easier.

Python 3.3 doesn't support HTTP/0.9 and doesn't have LineAndFileWrapper class.

This trivial "patch" solved the issue (reported back by user):
def _(self, *args):
 return self._readline()
httplib.LineAndFileWrapper._readline = httplib.LineAndFileWrapper.readline
httplib.LineAndFileWrapper.readline = _

I just noticed that this is reporducible consistently with the python
requests[1] module, if you route your request through a proxy. I was wondering
whether I should report this as a 'requests' bug or would this be the right
place to add a 'me too' ? Here's the reporducer:
[steve@localhost ~/venvs]$ mkdir requests_bug
[steve@localhost ~/venvs]$ cd requests_bug/
[steve@localhost ~/venvs/requests_bug]$ virtualenv .
New python executable in ./bin/python
Installing Setuptools..............................................................................................................................................................................................................................done.
Installing Pip.....................................................................................................................................................................................................................................................................................................................................done.
[steve@localhost ~/venvs/requests_bug]$ . bin/activate
(requests_bug)[steve@localhost ~/venvs/requests_bug]$ pip install requests
Downloading/unpacking requests
 Downloading requests-2.2.0.tar.gz (421kB): 421kB downloaded
 Running setup.py egg_info for package requests
Installing collected packages: requests
 Running setup.py install for requests
Successfully installed requests
Cleaning up...
(requests_bug)[steve@localhost ~/venvs/requests_bug]$ python
Python 2.7.5 (default, Nov 12 2013, 16:18:42)
[GCC 4.8.2 20131017 (Red Hat 4.8.2-1)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
You've got color, tab completion and pretty-printing. History will be saved
in /home/steve/.pyhistory when you exit.
Typing '\e' will open your $EDITOR with the last executed statement
>>> import requests
>>> requests.get('https://www.google.com/', verify=False)
<Response [200]>
>>> requests.get('https://www.google.com/', verify=False, proxies={'https':'https://192.168.117.157:443'})
Traceback (most recent call last):
 File "<console>", line 1, in <module>
 File "/home/steve/venvs/requests_bug/lib/python2.7/site-packages/requests/api.py", line 55, in get
 return request('get', url, **kwargs)
 File "/home/steve/venvs/requests_bug/lib/python2.7/site-packages/requests/api.py", line 44, in request
 return session.request(method=method, url=url, **kwargs)
 File "/home/steve/venvs/requests_bug/lib/python2.7/site-packages/requests/sessions.py", line 383, in request
 resp = self.send(prep, **send_kwargs)
 File "/home/steve/venvs/requests_bug/lib/python2.7/site-packages/requests/sessions.py", line 486, in send
 r = adapter.send(request, **kwargs)
 File "/home/steve/venvs/requests_bug/lib/python2.7/site-packages/requests/adapters.py", line 334, in send
 timeout=timeout
 File "/home/steve/venvs/requests_bug/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py", line 475, in urlopen
 conn = self._get_conn(timeout=pool_timeout)
 File "/home/steve/venvs/requests_bug/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py", line 217, in _get_conn
 return conn or self._new_conn()
 File "/home/steve/venvs/requests_bug/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py", line 656, in _new_conn
 return self._prepare_conn(conn)
 File "/home/steve/venvs/requests_bug/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py", line 625, in _prepare_conn
 conn.connect()
 File "/home/steve/venvs/requests_bug/lib/python2.7/site-packages/requests/packages/urllib3/connection.py", line 90, in connect
 self._tunnel()
 File "/usr/lib64/python2.7/httplib.py", line 759, in _tunnel
 line = response.fp.readline(_MAXLINE + 1)
TypeError: readline() takes exactly 1 argument (2 given)
>>>
cheers,
- steve
[1] requests-2.2.0

Requests may well want to put a workaround in place. To fix it in 2.7, we need a patch that actually respects the size argument, and a unit test.

I've been messing around trying to write a test for this, and it looks like genuinely the only place this can be hit is in _tunnel(). None of the other readline() calls can be hit with the LineAndFileWrapper() wrapping the file descriptor.

Alright, here's a patch.
I'm not ecstatic about this test: it's utterly contrived, but it also seems to be the least bad way to reproduce this bug. Let me know if you strongly object and I'll take another swing at it (it'll probably involve extending the FakeSocket quite extensively).

I made a similar patch today to fix the same issue, and I confirm the problem and the correctness of the solution
Please approve the patch and fix this bug.

I tried the patch, but the test fails for me:
test test_httplib failed -- Traceback (most recent call last):
 File "/home/rdmurray/python/p27/Lib/test/test_httplib.py", line 434, in test_proxy_tunnel_without_status_line
 conn.set_tunnel('foo')
 File "/home/rdmurray/python/p27/Lib/httplib.py", line 734, in set_tunnel
 raise RuntimeError("Can't setup tunnel for established connection.")
RuntimeError: Can't setup tunnel for established connection.

I ran into this problem when I gave https_proxy an invalid value:
 export https_proxy=http://foo.com
No divine intervention required. I was just trying to determine what message was printed with the proxy environment variable was set incorrectly.
Perhaps that will help you develop a more reasonable testcase.

Same error happens if http_proxy="223.19.230.181:80" variable is set.

This error occurs when using youtube-dl 
Complete log is here:
https://github.com/rg3/youtube-dl/issues/6727 

Left some review comments. But I don’t completely understand the test case. Is it true there are proxies that do not send a HTTP status line (like HTTP/1.0 200 Connection Established), but do sned the rest of the HTTP header section? This is what the test case seems to be modelling:
Client to proxy request:
b"CONNECT foo:80 HTTP/1.0\r\n"
b"\r\n"
Proxy (plus tunnelled?) response, minus a status line:
b"X-Foo: bar\r\n"
b"\r\n"
b"hello world"
It seems more likely that the proxies related to this bug are misbehaving, or there is some other network problem. Of the proxies mentioned in this report that are accessible to me, both of them do not appear to accept CONNECT requests at all. As soon as the first CONNECT line is send, a HTML blob saying "400 Bad Request" is received, without any HTTP header at all.

A test case with data like this would make more sense to me:
# Response without HTTP status line or header, modelling responses sent by
# some proxies that do not support CONNECT
body = b"<html>400 Bad Request</html>"

Martin: as noted in the comments on this issue, I believed the specific test case did not match any of the problems people were encountering: it was just the least bad way to trigger the specific flow that was being encountered.
In practice for these issues it's likely some other error condition is involved. Regardless, Python 2.7 claims to support HTTP/0.9, which certainly allows for proxies to not send a status line, so the test case is a reasonable example of what should be supported by httplib.
I'll take another run at this patch to see if I can clean it up.

Ok, new patch now attached.

I still don’t think the test case is a reasonable example. According to <http://www.w3.org/Protocols/HTTP/AsImplemented.html>, HTTP 0.9 only supported GET, not CONNECT, and doesn’t include any header fields in the protocol (such as your X-Foo: bar). It would make more sense to me to adjust the check at <https://hg.python.org/cpython/file/69ea73015132/Lib/httplib.py#l813> to also check for version == "HTTP/0.9". That would probably eliminate the need to mess too much with the details of the LineAndFileWrapper class.
But if you wanted to parse the CONNECT response as a HTTP 0.9 response, I think you should not try to parse any header, because that will tend to skip over the start of the actual response.
Also, I assume the change to "configure" is unrelated.

Martin, the idea of simply rejecting the LifeAndFileWrapper HTTP/0.9 fallback for _tunnel feels reasonable to me, that can work. Let me whip up an alternative patch that does that.

Ok, version three of the patch is now available.

Patch 3 looks better IMO. Now instead of the TypeError, we should see an error something like
socket.error: Tunnel connection failed: 200
I’ll commit this in a day or so, perhaps with an improved error message, like "Invalid response from tunnel request".

New changeset 3510e5d435db by Martin Panter <vadmium> in branch '2.7':
Issue #17849: Raise sensible exception for invalid HTTP tunnel response
https://hg.python.org/cpython/rev/3510e5d435db 

Thanks Cory. Error now looks like this:
$ https_proxy=http://223.19.230.181:80 ./python -c 'from urllib2 import *; urlopen("https://bugs.python.org/issue17849")'
[. . .]
urllib2.URLError: <urlopen error Invalid response from tunnel request>