While testing pywikibot using requests and urllib3 on Python 3.6 we got an interesting error:
======================================================================
ERROR: testQueryApiGetter (tests.wikidataquery_tests.TestApiSlowFunctions)
Test that we can actually retreive data and that caching works.
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/travis/build/xZise/pywikibot-core/tests/wikidataquery_tests.py", line 252, in testQueryApiGetter
data = w.query(q)
File "/home/travis/build/xZise/pywikibot-core/pywikibot/data/wikidataquery.py", line 601, in query
data = self.getDataFromHost(fullQueryString)
File "/home/travis/build/xZise/pywikibot-core/pywikibot/data/wikidataquery.py", line 563, in getDataFromHost
resp = http.fetch(url)
File "/home/travis/build/xZise/pywikibot-core/pywikibot/comms/http.py", line 359, in fetch
error_handling_callback(request)
File "/home/travis/build/xZise/pywikibot-core/pywikibot/comms/http.py", line 276, in error_handling_callback
raise request.data
File "/home/travis/build/xZise/pywikibot-core/pywikibot/comms/http.py", line 255, in _http_process
auth=auth, timeout=timeout, verify=True)
File "/home/travis/virtualenv/python3.6-dev/lib/python3.6/site-packages/requests/sessions.py", line 465, in request
resp = self.send(prep, **send_kwargs)
File "/home/travis/virtualenv/python3.6-dev/lib/python3.6/site-packages/requests/sessions.py", line 573, in send
r = adapter.send(request, **kwargs)
File "/home/travis/virtualenv/python3.6-dev/lib/python3.6/site-packages/requests/adapters.py", line 337, in send
conn = self.get_connection(request.url, proxies)
File "/home/travis/virtualenv/python3.6-dev/lib/python3.6/site-packages/requests/adapters.py", line 251, in get_connection
conn = self.poolmanager.connection_from_url(url)
File "/home/travis/virtualenv/python3.6-dev/lib/python3.6/site-packages/requests/packages/urllib3/poolmanager.py", line 139, in connection_from_url
return self.connection_from_host(u.host, port=u.port, scheme=u.scheme)
File "/home/travis/virtualenv/python3.6-dev/lib/python3.6/site-packages/requests/packages/urllib3/poolmanager.py", line 125, in connection_from_host
self.pools[pool_key] = pool
File "/home/travis/virtualenv/python3.6-dev/lib/python3.6/site-packages/requests/packages/urllib3/_collections.py", line 66, in __setitem__
_key, evicted_value = self._container.popitem(last=False)
KeyError: ('https', 'eu.wiktionary.org', 443)
Now that doesn't make much sense, as OrderedDict.popitem() only returns a KeyError if it's empty but then not with that error message. Unfortunately I can't reproduce it without doing the complete pywikibot test suite. When I only execute the tests where it occurred I don't get any failures. So I modified the output in urllib3 and returned the key as well as the content before popitem(last=False) is called (I shortened the values):
Key: ('https', 'bn.wikipedia.org', 443)
Content: OrderedDict([(('https', 'bn.wikipedia.org', 443), <requests.packages.urllib3.connectionpool.HTTPSConnectionPool object at 0x7fe483d991d0>), (('https', 'bs.wikipedia.org', 443), <...0x7fe483d99470>), (('https', 'ca.wikipedia.org', 443), <...0x7fe483da0e10>), (('https', 'cs.wikipedia.org', 443), <...0x7fe483dfc908>), (('https', 'da.wikipedia.org', 443), <...0x7fe483d98c88>), (('https', 'de.wikipedia.org', 443), <...0x7fe4911e06a0>), (('https', 'diq.wikipedia.org', 443), <...0x7fe483d6b400>), (('https', 'dsb.wikipedia.org', 443), <...0x7fe483d6bcf8>), (('https', 'en.wikipedia.org', 443), <...0x7fe483dc1da0>), (('https', 'eo.wikipedia.org', 443), <...0x7fe483d6cf28>), (('https', 'es.wikipedia.org', 443), <...0x7fe483d6b4a8>), (('https', 'fa.wikipedia.org', 443), <...0x7fe483d27278>), (('https', 'fi.wikipedia.org', 443), <...0x7fe483d6ce10>), (('https', 'fr.wikipedia.org', 443), <...0x7fe483e24cf8>), (('https', 'frr.wikipedia.org', 443), <...0x7fe483d39390>), (('https', 'ga.wikipedia.org', 443), <...0x7fe483c99668>), (('https', 'gl.wikipedia.org', 443), <...0x7fe483ca3f98>), (('https', 'als.wikipedia.org', 443), <...0x7fe483c4fcf8>), (('https', 'hu.wikipedia.org', 443), <...0x7fe483cd09e8>)])
As you can see it is not empty and the key in the KeyError is the first key in the OrderedDict. Also the key there is different from the key in the original exception I noticed so it's not a specific key that failed.
I don't think versions before Python 3.6 are affected as we had tests running on Python 3.5 (before Travis switched to 3.6 recently) and these all worked. Also not all popitem() calls in that line fail. I'm using Python 3.6.0a0 (default:d6c91b8242d2, Jul 18 2015, 16:36:01).
See also: https://github.com/shazow/urllib3/issues/680 and https://phabricator.wikimedia.org/T106212 |