homepage

PyUnicode_FSDecoder() accepts not only str and bytes or bytes-like object, but arbitrary iterable, e.g. list.
Example:
>>> compile('', [116, 101, 115, 116], 'exec')
<code object <module> at 0xb6fb1340, file "test", line 1>
I think accepting arbitrary iterables is unintentional and weird behavior.

I agree this doens't make sense.

I agree it is a bit strange. It looks like it is a victim of PyBytes_FromObject() doing more than it says; its documentation only mentions the buffer protocol, not accepting iterables.

PyUnicode_FSDecoder() is used in following functions in the stdlib:
compile()
symtable.symtable()
parser.compile()
parser.compilest()
zipimporter.zipimporter()
_imp.load_dynamic() (before 3.5)
This is behavior of PyUnicode_FSDecoder() from the start (issue9542). All above functions accepted only str in 3.1, thus accepting bytes object and others was new feature.
None tests are failed if reject non-str and non-bytes argument in PyUnicode_FSDecoder(). But none tests are failed even if disable support of bytes argument (there is a lack of tests for bytes path).
What should we do?
1. Add a warning when the argument neither str nor supporting the buffer protocol.
2. Drop support of non-str and not supporting the buffer protocol arguments without a warning.
3. Drop support of non-str and not supporting the buffer protocol arguments without a warning, and add a warning when the argument neither str nor bytes.
4. Drop support of non-str and non-bytes arguments without a warning.

See issue26800 for reasoning to go with #4

Proposed patch makes PyUnicode_FSDecoder() rejecting arbitrary iterables.

New changeset 2e48c2c4c733 by Serhiy Storchaka in branch '3.5':
Issue #26754: PyUnicode_FSDecoder() accepted a filename argument encoded as
https://hg.python.org/cpython/rev/2e48c2c4c733
New changeset e18ac7370113 by Serhiy Storchaka in branch 'default':
Issue #26754: PyUnicode_FSDecoder() accepted a filename argument encoded as
https://hg.python.org/cpython/rev/e18ac7370113 

Following patch deprecates the support of bytes-like objects (except bytes itself) in PyUnicode_FSDecoder() for consistency with issue26800.

New changeset 818f22f9ab02 by Serhiy Storchaka in branch 'default':
Issue #26754: Undocumented support of general bytes-like objects
https://hg.python.org/cpython/rev/818f22f9ab02