homepage

When running urllib2 and getting a BadStatus from an http server, this error is raised:
 File "/var/www/pinax-env/pline/apps/page/models.py", line 303, in render
 content = urllib2.urlopen(self.url,timeout=10).read()
 File "/usr/lib/python2.6/urllib2.py", line 124, in urlopen
 return _opener.open(url, data, timeout)
 File "/usr/lib/python2.6/urllib2.py", line 389, in open
 response = self._open(req, data)
 File "/usr/lib/python2.6/urllib2.py", line 407, in _open
 '_open', req)
 File "/usr/lib/python2.6/urllib2.py", line 367, in _call_chain
 result = func(*args)
 File "/usr/lib/python2.6/urllib2.py", line 1146, in http_open
 return self.do_open(httplib.HTTPConnection, req)
 File "/usr/lib/python2.6/urllib2.py", line 1119, in do_open
 r = h.getresponse()
 File "/usr/lib/python2.6/httplib.py", line 974, in getresponse
 response.begin()
 File "/usr/lib/python2.6/httplib.py", line 391, in begin
 version, status, reason = self._read_status()
 File "/usr/lib/python2.6/httplib.py", line 355, in _read_status
 raise BadStatusLine(line)
httplib.BadStatusLine
Because urllib2 doesn't catch it with this:
lines 1116-1120
try:
 h.request(req.get_method(), req.get_selector(), req.data, headers)
 r = h.getresponse()
except socket.error, err: # XXX what error?
 raise URLError(err)
It is not caught anywhere else and the call blows up unless I make a special exception for all httplib exceptions. The specific url that blew this up is:
http://phoenix.untd.com/oasx/rqst/type=sx/rdb=8203014d740000555355533a415a2d2d2d2d2d2d2d2d2d2d0100001d494a0901000000000000/version=3/origin=uol/isp=et_cau 

urllib2 is currently catching the socket.error exceptions and presenting the reason as custom URLError exception. To address this issue, the module should catch the httplib raised exceptions also and present it.

I've made a small change to urllib2 to catch the httplib.BadStatusLine and raise as URLError. This exception should rarely happen as it means the server is returning invalid responses. Nevertheless, I've added a test and hopefully fixed the issue.
Patch is attached. I will check if this needs to be added to newer versions of Python.

By the way, the issue can be recreated by running the following:
netcat -l -p 9999 -e "echo HTTP/1.1 1000 OK" &
python -c "import urllib2; urllib2.urlopen('http://localhost:9999')"
This happens on 2.6, 2.7 and 3 by the looks of it.

See also Issue 22797, about documenting that non-URLError exceptions may be raised.

Issue 13736 also proposes to wrap HTTP client exceptions, although I personally don’t really endorse this.
The other option is to fix the documentation: Issue 25633, Issue 22797.