homepage

I encountered this test failure while attempting to verify a patch for a separate issue, and I found that it occurs with the unmodified source on the default branch:
LD_LIBRARY_PATH="$PWD" ./python -bb -Wd -m test -r -w -uall -v test_curses
== CPython 3.3.0a4+ (default:4aeb5b9b62d7, Jun 8 2012, 10:23:35) [GCC 4.4.6
20110731 (Red Hat 4.4.6-3)]
== Linux-2.6.32-220.4.1.el6.x86_64-x86_64-with-centos-6.2-Final
little-endian
== /home/jbolling/cpython/build/test_python_26873
Testing with flags: sys.flags(debug=0, inspect=0, interactive=0, optimize=0,
dont_write_bytecode=0, no_user_site=0, no_site=0, ignore_environment=0,
verbose=0, bytes_warning=2, quiet=0, hash_randomization=1)
Using random seed 3072318
[1/1] test_curses
test test_curses crashed -- Traceback (most recent call last):
 File "/home/jbolling/cpython/Lib/test/regrtest.py", line 1237, in
runtest_inner
 test_runner()
 File "/home/jbolling/cpython/Lib/test/test_curses.py", line 338, in
test_main
 main(stdscr)
 File "/home/jbolling/cpython/Lib/test/test_curses.py", line 324, in main
 test_unget_wch(stdscr)
 File "/home/jbolling/cpython/Lib/test/test_curses.py", line 283, in
test_unget_wch
 read = chr(read)
OverflowError: signed integer is greater than maximum
1 test failed:
 test_curses
Re-running failed tests in verbose mode
Re-running test 'test_curses' in verbose mode
test test_curses crashed -- Traceback (most recent call last):
 File "/home/jbolling/cpython/Lib/test/regrtest.py", line 1237, in
runtest_inner
 test_runner()
 File "/home/jbolling/cpython/Lib/test/test_curses.py", line 338, in
test_main
 main(stdscr)
 File "/home/jbolling/cpython/Lib/test/test_curses.py", line 324, in main
 test_unget_wch(stdscr)
 File "/home/jbolling/cpython/Lib/test/test_curses.py", line 283, in
test_unget_wch
 read = chr(read)
OverflowError: signed integer is greater than maximum
[123272 refs]
Python was built and the tests run on CentOS 6.2 / x86_64, using the platform's standard tool chain, configured with "--enable-shared --with-threads --with-pydebug".
All other tests pass for me, although test_builtin failed when run as part of the whole suite but passed when run by itself.
For what it's worth, it looks like that particular message is emitted in exactly one place: Python/getargs.c:661 (function convertsimple()), which in this case I guess is being called indirectly from Python/bltinmodule.c:526 (function builtin_chr()). It's not obvious to me why that would be failing.

I just ran into this problem on another platform. I believe the problem is due to a sign-extension bug in the ncurses library unget_wch function (see link below). It was apparently fixed in nurses 5.8; I've tested with the current ncurses 5.9 and test_ncurses now runs without error.
Are you using a version of libncurses/libncursew older than 5.8? If so, can you try building with 5.8 or 5.9?
Another question is how to handle this for the 3.3.0 release. It looks like not all platforms have the latest ncurses and, if not, the unget_wch function and test_curses are likely to fail. Unfortunately, there doesn't seem to be an attribute available in the curses module that gives the version of the underlying curses/ncurses library. At the least, the problem should probably be documented somewhere.
http://invisible-island.net/ncurses/NEWS.html#t20091010 

The system on which I encountered the test failure uses ncurses 5.7, so that's consistent with the theory that the test is tickling an ncurses bug.
I'll have a look at testing with ncurses 5.8, but it is not available from RedHat or CentOS (and it never will be for the current and past versions of those systems), so that's not a good solution for most users.
On the other hand, it's not clear to me how serious is the bug revealed by the test failures, nor whether there is any viable workaround on the Python side.

Clarification: "so that's not a good solution for most users" ... of RedHat-family distros, version 6.2 and earlier.
In fact, it looks like RedHat is sticking with its current version of ncurses for RHEL 6.3, too, so no help is coming from that direction any time soon.

Ok, I confirm that the test passes after the system's ncurses library is upgraded to ncurses 5.8, and fails again when ncurses is downgraded back to version 5.7.

Thanks for the testing.
Georg, haypo: I think a call should be made on what, if anything, to do about this prior to 3.3.0-final. It seems that there are still OS distributions out there with older versions of ncurses. Is documenting this bug sufficient? If so, where?

I agree it should be documented, probably wherever unget_wch is documented.
If there is a way to detect the ncurses version, the test should be skipped on < 5.8 as well.

Documentation issue, can be resolved after b2.

To debug this issue, it would help to have the following information:
 - locale encoding: encoding variable
 - tested character: ch
 - character read by ncurses: read
Can someone reproducing the issue try to add: print("encoding=%s, ch=%r, read=%r" % (encoding, ch, read)) before the error?
It may be an issue in the Python implementation of unget_wch() or get_wch().
--
> I believe the problem is due to a sign-extension bug in the ncurses library unget_wch function (see link below).
Can we workaround this issue in Python? For example, use value & 0xffffff?
--
The test should be modified to use the encoding of stdscr, not the locale encoding: encoding = stdscr.encoding. (In this test, both encodings should be the same.)

New changeset 8282e4846e43 by Ned Deily in branch 'default':
Issue #15037: Build OS X installers with local copy of ncurses 5.9 libraries
http://hg.python.org/cpython/rev/8282e4846e43 

I've updated the OS X installers to build and link with a local copy of ncurses 5.9 rather than older Apple-supplied ones, thus avoiding the library bug. test_curses now passes for them.

Anything left to do here?

To be clear, I've eliminated the problem for the OS X installer builds by supplying newer versions of libncursesw so I wasn't planning on doing anything more on this issue myself. It should be easy enough to reproduce on most platforms by installing ncursesw 5.7.

New changeset e587426d719f by Ned Deily in branch 'default':
Issue #15037: Use correct path to system terminfo database.
http://hg.python.org/cpython/rev/e587426d719f 

Why is this blocking the release? It looks like a problem that can be solved in a bug fix release, or else by requiring that Python 3.3 users use a recent ncurses release.

This test still fails in Python 3.5 on Snow Leopard with the system ncurses; it would be nice to at least skip the test on systems with older ncurses.

Seems there is the same cause of the failure of test_curses on OpenBSD.
======================================================================
ERROR: test_window_funcs (test.test_curses.TestCurses) (meth='window.getkey')
Test the methods of windows
----------------------------------------------------------------------
Traceback (most recent call last):
 File "/home/serhiy/py/cpython3.7/Lib/test/test_curses.py", line 99, in test_window_funcs
 meth()
ValueError: chr() arg not in range(0x110000)
======================================================================
FAIL: test_unget_wch (test.test_curses.TestCurses)
----------------------------------------------------------------------
Traceback (most recent call last):
 File "/home/serhiy/py/cpython3.7/Lib/test/test_curses.py", line 355, in test_unget_wch
 self.assertEqual(read, ch)
AssertionError: -61 != 'é'
----------------------------------------------------------------------
System ncurses version is 5.7.

It is easy to write a workaround for the first case (but it is not ). The workaround for the second case would be too complex. I prefer to skip the test. Unfortunately the version of ncurses is not exposed on Python level, thus the skipping is OS-specific.

New changeset 7e68790f3db75a893d5dd336e6201a63bc70212b by Serhiy Storchaka in branch 'master':
bpo-15037: Add a workaround for getkey() in curses for ncurses 5.7 and earlier. (#3826)
https://github.com/python/cpython/commit/7e68790f3db75a893d5dd336e6201a63bc70212b

New changeset 1f81ea85e8e20347ec396001e5b869d36fe38398 by Serhiy Storchaka (Miss Islington (bot)) in branch '3.6':
bpo-15037: Add a workaround for getkey() in curses for ncurses 5.7 and earlier. (GH-3826) (#4218)
https://github.com/python/cpython/commit/1f81ea85e8e20347ec396001e5b869d36fe38398