homepage

I'm trying to fix IDLE to support Unicode (#4008 and #4323). Instead 
of IDLE builtin charset detection, I tried to use 
tokenize.detect_encoding() but this function doesn't work with script 
using Mac new line (b"\r").
Code to detect the encoding of a Python script:
----
def pythonEncoding(filename):
 with open(filename, 'rb') as fp:
 encoding, lines = detect_encoding(fp.readline)
 return encoding
----
Example to reproduce the problem with Mac script:
----
fp = BytesIO(b'# coding: ISO-8859-1\rprint("Bonjour ma ch\xe8re 
amie")\r')
encoding, lines = detect_encoding(fp.readline)
print(encoding, lines)
----
=> Result: utf-8 [b'# coding: ISO-8859-1\rprint("Bonjour ma ch\xe8re 
amie")\r']
The problem occurs at "line_string = line.decode('ascii')". 
Since "line" contains a non-ASCII character (b"\xe8"), the conversion 
fails.

See also related issue: #4628 (No universal newline support for 
compile() when using bytes).

I finally wrote a patch using a small generator based on the .splitlines(1) method. Patch includes a test.

Well, it looks like nobody cares (including me), so I close this issue.