homepage

I'm guessing this is a very seldom (never?) used code path. I've included a patch to test and fix the problem. The patch includes several related tests for pydoc.
$ py3 -c 'import pydoc; pydoc.synopsis("/opt/python3.4/lib/python3.4/lib-dynload/time.cpython-34m.so")'
Traceback (most recent call last):
 File "/opt/python3.4/lib/python3.4/tokenize.py", line 368, in find_cookie
 line_string = line.decode('utf-8')
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xf4 in position 96: invalid continuation byte
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
 File "<string>", line 1, in <module>
 File "/opt/python3.4/lib/python3.4/pydoc.py", line 229, in synopsis
 file = tokenize.open(filename)
 File "/opt/python3.4/lib/python3.4/tokenize.py", line 429, in open
 encoding, lines = detect_encoding(buffer.readline)
 File "/opt/python3.4/lib/python3.4/tokenize.py", line 409, in detect_encoding
 encoding = find_cookie(first)
 File "/opt/python3.4/lib/python3.4/tokenize.py", line 373, in find_cookie
 raise SyntaxError(msg)
SyntaxError: invalid or missing encoding declaration for '/opt/python3.4/lib/python3.4/lib-dynload/time.cpython-34m.so'

FYI, I found this while working on issue #19703.

Well please get a second opinion. I don't know why you added me, I'm not qualified.

Not sure why I nosy'ed you, either. :)

In 2.7 the code just does an open. Victor changed it to call tokenize.open in 3.2, but tokenize.open obviously only works on python source files. So the logic of that method is now completely wrong.
I'm not sure the logic made a lot of sense even before...if the extension is binary it just closes it again. So the initial open is for its side effect of returning None if the file can't be opened.
Anyway, the logic of that method clearly needs to be rewritten. And obviously there are some missing tests....
I don't see any patch attached to the issue, by the way.

Thanks for taking a look. Here's the patch that I totally forget to attach.

Is it the case that given a filename, it might be possible to load a module even if open(filename) fails?
I think the logic is clearer in the form where it is not pulled out into a separate helper function. You can avoid the double check on the extension by doing:
 if filename.endswith(importlib.machinery.BYTECODE_SUFFIXES):
 loader = importlib.machinery.SourcelessFileLoader('__temp__',
 filename)
 elif filename.endswith(importlib.machinery.EXTENSION_SUFFIXES):
 loader = importlib.machinery.ExtensionFileLoader('__temp__',
 filename)
 else:
 loader = None
 if loader:
 xxxxx
 else:
 xxxxx

Sounds good to me. Here's an updated patch.

Looks good, except you can replace those any calls with simple calls to endswith using a tuple of strings (I forgot the call to tuple in my example).

New changeset d6c3fb8d5f84 by Eric Snow in branch 'default':
Issue 20123: Fix pydoc.synopsis() for "binary" modules.
http://hg.python.org/cpython/rev/d6c3fb8d5f84 

Ah, I missed that in your earlier suggestion. I followed your recommendation. Thanks for that. :)

This broke one of the FreeBSD buildbots:
http://buildbot.python.org/all/builders/AMD64%20FreeBSD%209.0%203.x/builds/6102 

New changeset ff3be21338d5 by Eric Snow in branch 'default':
Issue 20123: try using a different builtin module in a pydoc test.
http://hg.python.org/cpython/rev/ff3be21338d5 

New changeset efcf163d04f5 by Eric Snow in branch 'default':
Issue 20123: Disable a problematic test.
http://hg.python.org/cpython/rev/efcf163d04f5 

I've run out of time to trouble-shoot the failure (specific to 1 buildbot). Until I can get back to it, I've disabled the problematic test (even though it's only a problem on 1 buildbot).

The buildbot is happy again. I'll address fixing that test in issue #20128.

Thanks for seeing this through, Eric.

I realized today that this should probably be fixed in 3.3 as well, as RDM implied by marking it as also a 3.3 bug. :)

koobs-freebsd9 (3.4) buildbot has also been failing for a while on what seems to be this changeset:
======================================================================
ERROR: test_synopsis_sourceless (test.test_pydoc.PydocDocTest)
----------------------------------------------------------------------
Traceback (most recent call last):
 File "/usr/home/buildbot/python/3.4.koobs-freebsd9/build/Lib/test/test_pydoc.py", line 504, in test_synopsis_sourceless
 synopsis = pydoc.synopsis(filename)
 File "/usr/home/buildbot/python/3.4.koobs-freebsd9/build/Lib/pydoc.py", line 238, in synopsis
 mtime = os.stat(filename).st_mtime
FileNotFoundError: [Errno 2] No such file or directory: '/usr/home/buildbot/python/3.4.koobs-freebsd9/build/Lib/__pycache__/os.cpython-34.pyc'
Full buildlog is attached.

fyi - tracking the new issue koobs reported in http://bugs.python.org/issue22910 

Re-enabling the test is tracked on 20128, so this issue is no longer needed.