homepage

IDLE checksyntax() function doesn't support Unicode. Example with 
idle-3.0rc1-quits-when-run.py in an ASCII terminal:
$ ./python Tools/scripts/idle
Exception in Tkinter callback
Traceback (most recent call last):
 File "/home/haypo/prog/py3k/Lib/tkinter/__init__.py", line 1405, in 
__call__
 return self.func(*args)
 File "/home/haypo/prog/py3k/Lib/idlelib/ScriptBinding.py", line 124, 
in run_module_event
 code = self.checksyntax(filename)
 File "/home/haypo/prog/py3k/Lib/idlelib/ScriptBinding.py", line 86, 
in checksyntax
 source = f.read()
 File "/home/haypo/prog/py3k/Lib/io.py", line 1719, in read
 decoder.decode(self.buffer.read(), final=True))
 File "/home/haypo/prog/py3k/Lib/io.py", line 1294, in decode
 output = self.decoder.decode(input, final=final)
 File "/home/haypo/prog/py3k/Lib/encodings/ascii.py", line 26, in 
decode
 return codecs.ascii_decode(input, self.errors)[0]
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 
87: ordinal not in range(128)
To open an ASCII terminal on Linux, you can for example use xterm with 
an empty environment (except DISPLAY and HOME variables): "env -i 
DISPLAY=$DISPLAY HOME=$HOME xterm".

Hum, the problem is that IDLE asks io.open() to detect the charset 
whereas open() doesn't know the #coding: header. So if your locale is 
ASCII, CP1252 or anything different of UTF-8, read the file will 
fails.
I wrote a patch to detect the encoding. Python code (detect_encoding() 
function) is based on PyTokenizer_FindEncoding() and get_coding_spec() 
(from Parser/tokenizer.c). There is no existing Python function to 
detect the encoding of a Python script? (a public function available 
in a Python script)

Ah! tokenize has already a method detect_encoding(). My new patch uses 
it to avoid code duplication.

Notice that there is also IOBinding.coding_spec. Not sure whether this
or the one in tokenize is more correct.

loewis wrote:
> Notice that there is also IOBinding.coding_spec.
> Not sure whether this or the one in tokenize is more correct.
Oh! IOBinding reimplement many features now available in Python like 
universal new line or function to write unicode strings to a file. But 
I don't want to rewrite IDLE, I just want to fix the initial problem: 
IDLE is unable to open a non-ASCII file using "#coding:" header.
So IDLE reimplemented coding detection twice: once in IOBinding and 
once in ScriptBinding. So I wrote a new version of my patch removing 
all the code to reuse tokenize.detect_encoding().
I changed IDLE behaviour: IOBinding._decode() used the locale encoding 
if it's unable to detect the encoding using UTF-8 BOM and/or if the 
#coding: header is missing. Since I also read "Finally, try the 
locale's encoding. This is deprecated", I prefer to remove it. If you 
want to keep the current behaviour, use:
-------------------------
def detect_encoding(filename, default=None):
 with open(filename, 'rb') as f:
 encoding, line = tokenize.detect_encoding(f.readline)
 if (not line) and default:
 return default
 return encoding
...
 encoding = detect_encoding(filename, locale_encoding)
-------------------------
Please review and test my patch (which becomes longer and longer) :-)

> Oh! IOBinding reimplement many features now available in Python like 
> universal new line or function to write unicode strings to a file.
It did not *re*implement. The implementation in IOBinding predates all
other implementations out there.

@loewis: Ok, I didn't know. I think that it's better to reuse existing 
code.
I also compared the implementations of encoding detection, and the 
code looks the same in IDLE and tokenize, but I prefer tokenize. 
tokenize.detect_encoding() has longer documentation, return the line 
(decoded as Unicode) matching the encoding cookie, and look to be more 
robust.
I saw an interesting test in IDLE code: it checks the charset. So I 
wrote a patch raising a SyntaxError for tokenize: issue4021.

I can't reproduce the problem. It works fine for me, displaying the box
drawing character. In case it matters, sys.getpreferredencoding returns
'ANSI_X3.4-1968'; this is on Linux, idle started from an xterm, r66761 

@loewis: I guess that your locale is still UTF-8.
On Linux (Ubuntu Gutsy) using "env -i DISPLAY=$DISPLAY HOME=$HOME 
xterm" to get a new empty environment, I get:
$ locale
LANG=
LC_ALL=
LC_CTYPE="POSIX"
LC_NUMERIC="POSIX"
LC_TIME="POSIX"
LC_COLLATE="POSIX"
...
$ python3.0
>>> from idlelib.IOBinding import encoding
>>> encoding 
'ansi_x3.4-1968'
>>> import locale
>>> locale.getdefaultlocale()
(None, None)
>>> locale.nl_langinfo(locale.CODESET)
'ANSI_X3.4-1968'
In this environment, IDLE is unable to detect 
idle-3.0rc1-quits-when-run.py encoding.
IDLE uses open(filename, 'r'): it doesn't specify the charset. In this 
case, TextIOWrapper uses locale.getpreferredencoding() as encoding (or 
ASCII on failure).
To sum IDLE: if your locale is UTF-8, you will be able to open an 
UTF-8 file. So for example, if your locale is UTF-8, you won't be able 
to open an ISO-8859-1 file. Let's try iso.py: IDLE displays the 
error "Failed to decode" and quit whereas I specified the encoding :-/

> @loewis: I guess that your locale is still UTF-8.
To refute this claim, I reported that locale.getpreferredencoding
reports 'ANSI_X3.4-1968'. I was following your instructions exactly
(on Debian 4.0), and still, it opens successfully (when loaded through
File/Open). Should I do something else with it to trigger the error,
other than opening it?
When opening iso.py, I get a pop window titled "Decoding error",
with a message "Failed to Decode". This seems to be correct also.
So I still can't reproduce the problem.
I don't understand why you say that IDLE uses open(filename, 'r').
In IOBinding.IOBinding.loadfile, I see
 # open the file in binary mode so that we can handle
 # end-of-line convention ourselves.
 f = open(filename,'rb')

IDLE opens the script many than once. There are two cases:
 (1) first open when IDLE read the file content to display it
 (2) second open on pressing F5 key (Run Module) to check the syntax
(1) uses IOBinding and fails to open ISO-8859-1 file with UTF-8 
locale.
(2) uses ScriptBinding and fails to open UTF-8 file with ASCII locale.
About the initial problem (idle-3.0rc1-quits-when-run.py), yes, I 
forgot to say that you have to run the module, sorry :-/

This patch has two problems:
1. saving files fails, since there is still a call left to the function
coding_spec, but that function is removed.
2. if saving would work: it doesn't preserve the line endings of the
original file when writing it back. If you open files with DOS line
endings on Unix, upon saving, they should still have DOS line endings.

This is still a problem on my WinXP 3.0rc3 with
# -*- coding: utf-8 -*-
in a file but not with the same pasted directly into the shell Window.

Here is a new patch that fixes this issue, and the duplicate issues
(#4410, and #4623).
It doesn't try to eliminate code duplication, but fixes coding_spec by
decoding always to Latin-1 first until the coding is known. It fixes
check_syntax by opening the source file in binary. It should have fixed
tabnanny the same way, except that tabnanny cannot properly process byte
tokens.

I vote for fixing this too. This might be simplified/another example of
above mentioned issues:
# -*- coding: utf-8 -*-
print ("ěščřžýáíé")
in IDLE prints this:
>>> 
ěščĹTMžà ̋áíé
When running this script under python command line from another editor,
I get the output readable as expected.

Committed as r68730 and r68731.