homepage

I'm not sure what the best solution for this is. If I truncate the
header values at a \x00 character it ends in an infinite redirect loop
(which urllib2 detects and raises on). If I simple remove all \x00
characters the resulting url is not accepted by wikispaces.com due to
having an extra / in it.
Verdict: wikispaces.com is broken.
urllib2 could do better. wget and firefox deal with it properly. but
i'll leave deciding which patch to use up to someone who cares about
handling broken sites.
patch to implement either behavior of dealing with nulls where they
shouldn't be:
Index: Lib/httplib.py
===================================================================
--- Lib/httplib.py (revision 62033)
+++ Lib/httplib.py (working copy)
@@ -291,9 +291,18 @@
 break
 headerseen = self.isheader(line)
 if headerseen:
+ # Some bad web servers reply with headers with a \x00 null
+ # embedded in the value. Other http clients deal with
+ # this by treating it as a value terminator, ignoring the
+ # rest so we will too. http://bugs.python.org/issue2464.
+ if '\x00' in line:
+ line = line[:line.find('\x00')]
+ # if you want to just remove nulls instead use this:
+ #line = line.replace('\x00', '')
 # It's a legal header line, save it.
 hlist.append(line)
- self.addheader(headerseen,
line[len(headerseen)+1:].strip())
+ value = line[len(headerseen)+1:].strip()
+ self.addheader(headerseen, value)
 continue
 else:
 # It's not a header line; throw it back and stop here.