homepage

http://www.python.org/dev/buildbot/3.x/builders/x86 Tiger 3.x/builds/6/steps/test/logs/stdio
test test_pep277 failed -- multiple errors occurred; run in verbose mode for details
Re-running test test_pep277 in verbose mode
test_directory (test.test_pep277.UnicodeFileTests) ... ok
test_failures (test.test_pep277.UnicodeFileTests) ... ok
test_listdir (test.test_pep277.UnicodeFileTests) ... FAIL
test_normalize (test.test_pep277.UnicodeFileTests) ... ERROR
test_open (test.test_pep277.UnicodeFileTests) ... ok
test_rename (test.test_pep277.UnicodeFileTests) ... ok
test_directory (test.test_pep277.UnicodeNFCFileTests) ... ok
test_failures (test.test_pep277.UnicodeNFCFileTests) ... ok
test_listdir (test.test_pep277.UnicodeNFCFileTests) ... FAIL
test_normalize (test.test_pep277.UnicodeNFCFileTests) ... ERROR
test_open (test.test_pep277.UnicodeNFCFileTests) ... ok
test_rename (test.test_pep277.UnicodeNFCFileTests) ... ok
test_directory (test.test_pep277.UnicodeNFDFileTests) ... ok
test_failures (test.test_pep277.UnicodeNFDFileTests) ... ok
test_listdir (test.test_pep277.UnicodeNFDFileTests) ... ok
test_normalize (test.test_pep277.UnicodeNFDFileTests) ... ERROR
test_open (test.test_pep277.UnicodeNFDFileTests) ... ok
test_rename (test.test_pep277.UnicodeNFDFileTests) ... ok
test_directory (test.test_pep277.UnicodeNFKCFileTests) ... ok
test_failures (test.test_pep277.UnicodeNFKCFileTests) ... ok
test_listdir (test.test_pep277.UnicodeNFKCFileTests) ... FAIL
test_normalize (test.test_pep277.UnicodeNFKCFileTests) ... ERROR
test_open (test.test_pep277.UnicodeNFKCFileTests) ... ok
test_rename (test.test_pep277.UnicodeNFKCFileTests) ... ok
test_directory (test.test_pep277.UnicodeNFKDFileTests) ... ok
test_failures (test.test_pep277.UnicodeNFKDFileTests) ... ok
test_listdir (test.test_pep277.UnicodeNFKDFileTests) ... ok
test_normalize (test.test_pep277.UnicodeNFKDFileTests) ... ERROR
test_open (test.test_pep277.UnicodeNFKDFileTests) ... ok
test_rename (test.test_pep277.UnicodeNFKDFileTests) ... ok
======================================================================
ERROR: test_normalize (test.test_pep277.UnicodeFileTests)
----------------------------------------------------------------------
test test_pep277 crashed -- <class 'UnicodeEncodeError'>: 'ascii' codec can't encode characters in position 222-239: ordinal not in range(128)
Traceback (most recent call last):
 File "./Lib/test/regrtest.py", line 905, in runtest_inner
 File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/test/test_pep277.py", line 195, in test_main
 UnicodeNFKDFileTests,
 File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/test/support.py", line 1000, in run_unittest
 _run_suite(suite)
 File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/test/support.py", line 974, in _run_suite
 result = runner.run(suite)
 File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/unittest/runner.py", line 158, in run
 result.printErrors()
 File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/unittest/runner.py", line 108, in printErrors
 self.printErrorList('ERROR', self.errors)
 File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/unittest/runner.py", line 116, in printErrorList
 self.stream.writeln("%s" % err)
 File "/Users/db3l/buildarea/3.x.bolen-tiger/build/Lib/unittest/runner.py", line 24, in writeln
 self.write(arg)
UnicodeEncodeError: 'ascii' codec can't encode characters in position 222-239: ordinal not in range(128)

See also #8422.

Some diagnosis information ...
 - happens for 2 tests inside test_pep277: test_listdir and test_normalize
 - happens only for 4 weird filenames (added with #8180)
 - happens only on "x86 Tiger" buildbot ("PPC Tiger" does not report this error)
I've added some lines to regrtest.py to identify each platform:
"x86 Tiger": Darwin-8.10.1-i386-32bit little-endian (OS X 10.4.10)
"PPC Tiger": Darwin-8.11.0-Power_Macintosh-powerpc-32bit big-endian (OS X 10.4.11)
I don't know how the endianness may be linked with this issue.
Failing files (4):
 u'\u0385\u03d3\u03d4',
 u'\u00a8\u0301\u03d2\u0301\u03d2\u0308', # == NFD(u'\u0385\u03d3\u03d4')
 u'\u0020\u0308\u0301\u038e\u03ab', # == NFKC(u'\u0385\u03d3\u03d4')
 u'\u1e9b\u1fc1\u1fcd\u1fce\u1fcf\u1fdd\u1fde\u1fdf\u1fed',
One solution is to skip these files on all 'darwin' platforms, without losing more time on this issue.

It prefer if it were clear why these tests fail (but don't have time to look into this at the moment). That is, I don't mind if the tests get disabled on darwin, but only if there is a documented reason for it because otherwise disabling the test might hide a real issue.

> It prefer if it were clear why these tests fail
I agree. There's little point having tests if we're just going to disable them when they fail without understanding why they're failing first.
At that point a green buildbot just means "Hey, we disabled all failing tests!", rather than being an indication that Python's working well on that platform.

The gory details...
>>> import os, unicodedata
>>> tempdir = u'temp_pep277'
>>> name = u'\u0385\u03d3\u03d4'
>>> os.mkdir(tempdir)
>>> with open(os.path.join(tempdir, name), 'w') as f:
... f.write('foo')
>>> unicodedata.normalize('NFD', name)
u'\xa8\u0301\u03d2\u0301\u03d2\u0308'
>>> os.listdir(tempdir)
[u'\xa8\u0301\u03d2\u03d2\u0301\u0308']
The filename should use the Normal Decomposition (NFD) of the original filename. But the decomposition looks wrong on this platform.
The problem is similar for the other special filenames. Examples:
- u'\u0385\u03d3\u03d4'
 ==> NFD: u'\xa8\u0301\u03d2\u0301\u03d2\u0308'
 ==> OSX: u'\xa8\u0301\u03d2\u03d2\u0301\u0308'
- u' \u0308\u0301\u038e\u03ab'
 ==> NFD: u' \u0308\u0301\u03a5\u0301\u03a5\u0308'
 ==> OSX: u' \u0308\u0301\u03a5\u03a5\u0301\u0308'
- u'\u1e9b\u1fc1\u1fcd\u1fce\u1fcf\u1fdd\u1fde\u1fdf\u1fed'
 ==> NFD: u'\u017f\u0307\xa8\u0342\u1fbf\u0300\u1fbf\u0301\u1fbf\u0342\u1ffe\u0300\u1ffe\u0301\u1ffe\u0342\xa8\u0300'
 ==> OSX: u'\u017f\u0307\xa8\u0342\u1fbf\u0300\u1fbf\u0301\u1fbf\u1ffe\u1ffe\u1ffe\xa8\u0342\u0300\u0301\u0342\u0300'
I have no idea how to investigate further.
And the "PPC Tiger" buildbot does not fail here.
Reference about OSX decomposition:
http://developer.apple.com/mac/library/qa/qa2001/qa1173.html 

> Reference about OSX decomposition:
> http://developer.apple.com/mac/library/qa/qa2001/qa1173.html
Mac OS X source code:
http://www.opensource.apple.com/source/xnu/xnu-1504年3月12日/bsd/vfs/vfs_utfconv.c?txt 

#8207 was the same issue and r79426 (for trunk and r79426 in py3k) was supposed to fix it.

I opened the issue #10209: "Mac OS X: Decompose filenames on encode, and precompose filenames on decode".

> I opened the issue #10209: "Mac OS X: Decompose filenames on encode, 
> and precompose filenames on decode".
It was a bad idea. I fixed test_pep277 instead with:
 - r85897 disables the filenames that are normalized differently by Python and by darwin
 - r85899 disables test_normalize and test_listdir tests
And test_pep277 now pass on "x86 Tiger 3.x" buildbot, and so I can close this issue (and issue #10209).