homepage

A simple reproducer:
python -c 'import readline' | xxd
0000000: 1b5b 3f31 3033 3468 .[?1034h
This was reported at [1] and originally at [2]. The readline maintainer suggests [3] using:
rl_variable_bind ("enable-meta-key", "off");
which was introduced in readline 6.1. Do you think it'd be safe to add the above line?
Thanks.
[1] https://bugzilla.redhat.com/show_bug.cgi?id=880393
[2] https://bugzilla.redhat.com/show_bug.cgi?id=593799
[3] http://lists.gnu.org/archive/html/bug-readline/2011-04/msg00009.html 

I can't reproduce under Ubuntu 13.10. Is this Red Hat-specific?
(according to Dave, """This is a readline bug; it looks like it should not emit those characters when stdout is not a tty""")

Reproducible under Fedora 18.
$ ./python -c "import readline" | hexdump -C
00000000 1b 5b 3f 31 30 33 34 68 |.[?1034h|
00000008
$ TERM=dumb ./python -c "import readline" | hexdump -C

I can also reproduce it on Arch Linux. It seems that the bad characters are only output if env variable TERM starts with "xterm".

We seem to have gotten bit by this in issue21425.
Dave, is it possible for you to validate the proposed fix? I can't test this under Ubuntu.
Setting priority to "high" as it seems to actually annoy many people:
http://stackoverflow.com/questions/15760712/python-readline-module-prints-escape-character-during-import
http://reinout.vanrees.org/weblog/2009/08/14/readline-invisible-character-hack.html
http://lists.gnu.org/archive/html/bug-readline/2007-08/msg00004.html 

Attached readline_disable_meta_key.patch: Implement the workaround suggested in (*), but only use the workaround if stdout is not a TTY (ex: output redirected), to limit the risk of regression.
(*) http://lists.gnu.org/archive/html/bug-readline/2011-04/msg00009.html
Extract of the patch:
+ if (!isatty(STDOUT_FILENO)) {
+ /* Issue #19884: Don't try to enable any meta modifier key the terminal
+ claims to support when it is called. On many terminals (ex:
+ xterm-256color), the meta key is used to send eight-bit characters
+ (ANSI sequence "033円[1034h"). */
+ rl_variable_bind ("enable-meta-key", "off");
+ }
This issue becomes very annoying on my Fedora 20. The output of any Mercurial command now starts with "033円.[?1034h" (Mercurial uses Python 2.7). Example:
haypo@smithers$ hg root|hexdump -C
00000000 1b 5b 3f 31 30 33 34 68 2f 68 6f 6d 65 2f 68 61 |.[?1034h/home/ha|
00000010 79 70 6f 2f 70 72 6f 67 2f 70 79 74 68 6f 6e 2f |ypo/prog/python/|
00000020 64 65 66 61 75 6c 74 0a |default.|
00000028
Fedora 18 changed the default TERM environment variable to "xterm-256color":
http://fedoraproject.org/wiki/Features/256_Color_Terminals
Workaround in your application (to run on unpatched Python): set the TERM environment variable to "dummy", or unset this variable.

New changeset 0177d8a4e82a by Victor Stinner in branch '2.7':
Issue #19884: readline: Disable the meta modifier key if stdout is not a
http://hg.python.org/cpython/rev/0177d8a4e82a
New changeset 6303266beb80 by Victor Stinner in branch '3.4':
Issue #19884: readline: Disable the meta modifier key if stdout is not a
http://hg.python.org/cpython/rev/6303266beb80
New changeset f85a968f9e01 by Victor Stinner in branch 'default':
(Merge 3.4) Issue #19884: readline: Disable the meta modifier key if stdout is
http://hg.python.org/cpython/rev/f85a968f9e01 

I commited my patch.

The test fails on AMD64 OpenIndiana 2.7:
http://buildbot.python.org/all/builders/AMD64%20OpenIndiana%202.7/builds/2354/steps/test/logs/stdio
test test_readline failed -- Traceback (most recent call last):
 File "/export/home/buildbot/64bits/2.7.cea-indiana-amd64/build/Lib/test/test_readline.py", line 52, in test_init
 self.assertEqual(stdout, b'')
AssertionError: '\x1b[?1034h' != ''

The changes are also causing segfaults when readline is linked with BSD libedit (the default on OS X) rather than GNU readline:
======================================================================
FAIL: test_init (test.test_readline.TestReadline)
----------------------------------------------------------------------
Traceback (most recent call last):
 File "/py/dev/3x/source/Lib/test/test_readline.py", line 52, in test_init
 TERM='xterm-256color')
 File "/py/dev/3x/source/Lib/test/script_helper.py", line 69, in assert_python_ok
 return _assert_python(True, *args, **env_vars)
 File "/py/dev/3x/source/Lib/test/script_helper.py", line 55, in _assert_python
 "stderr follows:\n%s" % (rc, err.decode('ascii', 'ignore')))
AssertionError: Process return code is -11, stderr follows:
Fatal Python error: Segmentation fault
Current thread 0x00007fff75489310 (most recent call first):
 File "<frozen importlib._bootstrap>", line 321 in _call_with_frames_removed
 File "<frozen importlib._bootstrap>", line 1664 in load_module
 File "<frozen importlib._bootstrap>", line 540 in _check_name_wrapper
 File "<frozen importlib._bootstrap>", line 1110 in _load_backward_compatible
 File "<frozen importlib._bootstrap>", line 1140 in _load_unlocked
 File "<frozen importlib._bootstrap>", line 2175 in _find_and_load_unlocked
 File "<frozen importlib._bootstrap>", line 2186 in _find_and_load
 File "<string>", line 1 in <module>

> The changes are also causing segfaults when readline is linked with BSD libedit (the default on OS X) rather than GNU readline:
Oh wow. Do you have an idea of to fix the issue with libedit? Or make
the code condition, only use it with native readline?

Currently, readline.c uses #ifdef __APPLE__ to guard libedit-specific code (there is another open issue to generalize libedit support to other platforms).

New changeset f0ab6f9f0603 by Victor Stinner in branch '2.7':
Issue #19884, readline: calling rl_variable_bind ("enable-meta-key", "off")
http://hg.python.org/cpython/rev/f0ab6f9f0603
New changeset 3f08c1156050 by Victor Stinner in branch '3.4':
Issue #19884, readline: calling rl_variable_bind ("enable-meta-key", "off")
http://hg.python.org/cpython/rev/3f08c1156050
New changeset 0ed1801bf4bd by Victor Stinner in branch 'default':
(Merge 3.4) Issue #19884, readline: calling rl_variable_bind
http://hg.python.org/cpython/rev/0ed1801bf4bd 

The test fails also on OpenBSD:
http://buildbot.python.org/all/builders/x86%20OpenBSD%205.5%203.x/builds/671/steps/test/logs/stdio
======================================================================
FAIL: test_init (test.test_readline.TestReadline)
----------------------------------------------------------------------
Traceback (most recent call last):
 File "/home/python-builds/3.x.borja-openbsd-x86/build/Lib/test/test_readline.py", line 53, in test_init
 self.assertEqual(stdout, b'')
AssertionError: b'\x1b[?1034h' != b''

Oh, the test also fails on the new Red Hat 6 buildbot:
http://buildbot.python.org/all/builders/x86%20RHEL%206%203.x/builds/4358/steps/test/logs/stdio
======================================================================
FAIL: test_init (test.test_readline.TestReadline)
----------------------------------------------------------------------
Traceback (most recent call last):
 File "/home/buildbot/buildarea/3.x.coghlan-redhat/build/Lib/test/test_readline.py", line 53, in test_init
 self.assertEqual(stdout, b'')
AssertionError: b'\x1b[?1034h' != b''

This problem shows up on OpenBSD too. It breaks 'hg view' also.

It seems this bug was fixed properly by readline in version 6.3; rl_initialize won't put meta codes on the screen. Frankly, I'm surprised distros like Fedora haven't upgraded or patched readline themselves to fix this. Aren't other programs affected?
Turning off "enable-meta-key" isn't great, since it doesn't work on older readline or libedit, and is probably technically a backwards compatibility problem.
I think the best solution would be not call rl_initialize during the initialization of the readline module, but I'm not sure if that could create compatibility problems.

Should this be a release blocker for 3.4.2?

The test fails on some platforms but it's not a regression.

Issue22773 provides a patch to export Readline version and skip the test for earlier releases of libreadline where turning off enable-meta-key does not work.

New changeset c4b5a5d44254 by Antoine Pitrou in branch '3.4':
Issue #22773: fix failing test with old readline versions due to issue #19884.
https://hg.python.org/cpython/rev/c4b5a5d44254
New changeset be374b8c40c8 by Antoine Pitrou in branch 'default':
Issue #22773: fix failing test with old readline versions due to issue #19884.
https://hg.python.org/cpython/rev/be374b8c40c8 

New changeset eba6e68e818c by Antoine Pitrou in branch '2.7':
Issue #22773: fix failing test with old readline versions due to issue #19884.
https://hg.python.org/cpython/rev/eba6e68e818c 

Arfever, Antoine: If buildbots are happy (green), you can close the issue. (I'm answering to your question on IRC ;-))

I think the version check (`readline._READLINE_VERSION < 0x0600`) is incorrect, as the test still fails for me on Mac OS X 10.9 with readline version 6.2 (0x602). Upgrading to 6.3 (0x603) fixes it, though.

Mac OS X uses libedit, not libreadline.

From the OP:
> This was reported at [1] and originally at [2]. The readline maintainer suggests [3] using:
> 
> rl_variable_bind ("enable-meta-key", "off");
> 
> which was introduced in readline 6.1. Do you think it'd be safe to add the above line?
From 3.4.3 final:
> @unittest.skipIf(readline._READLINE_VERSION < 0x0600
> and "libedit" not in readline.__doc__,
> "not supported in this library version")
The test currently fails on readline version 6.0. The version to test on needs a bump to 0x0610.

Whoops, that's 0x0601. Though Maxime gives evidence that the version should in fact be 0x0603. (Note that while OS X ships with libedit over libreadline, anyone who wants to can install the real thing instead of that pale imitation; the test would have been skipped if Maxime were using libedit.)

The version check doesn't work because 0ed1801bf4bd added #ifndef __APPLE__ to guard the code, so if you're using readline on OS X, that rl_variable_bind workaround won't work.
There are two alternatives:
1) convert ifndef to ifdef and add a check for the ``using_libedit_emulation`` variable.
 #ifdef __APPLE__
 if (using_libedit_emulation) {
 ...
 ...
looks like a common idiom in readline.c.
2) just skip the test if _READLINE_VERSION < 0x0603

I tried applying this patch and its failing and throwing the following error:
patching file Lib/test/test_readline.py
Hunk #1 FAILED at 1.
Hunk #2 FAILED at 43.
2 out of 2 hunks FAILED -- saving rejects to file Lib/test/test_readline.py.rej
patching file Modules/readline.c
Hunk #1 FAILED at 1019.
1 out of 1 hunk FAILED -- saving rejects to file Modules/readline.c.rej
I'm using Mac Os El Captain

Yogesh: Victor’s patch has already been applied. What is left to do is another patch that enables Victor’s code on OS X when Gnu Readline is being used, as opposed to the usual Apple Editline.
Also, I think it is valid to update the version check to 6.1. According to <https://cnswww.cns.cwru.edu/php/chet/readline/CHANGES>, enable-meta-key is in 6.1, but not 6.0. On 6.0 the code probably has no effect.

Here is a patch to enable the workaround on OS X, and to adjust the test condition to 6.1. It would be nice if someone with OS X and Gnu Readline can confirm that this fixes the problem.

I'm still seeing this test failure on EL6.8 with python 3.4.5:
[268/391] test_readline
testHistoryUpdates (test.test_readline.TestHistoryManipulation) ... ok
test_init (test.test_readline.TestReadline) ... test test_readline failed
FAIL
======================================================================
FAIL: test_init (test.test_readline.TestReadline)
----------------------------------------------------------------------
Traceback (most recent call last):
 File "/builddir/build/BUILD/Python-3.4.5/Lib/test/test_readline.py", line 57, in test_init
 self.assertEqual(stdout, b'')
AssertionError: b'\x1b[?1034h' != b''
----------------------------------------------------------------------
Ran 2 tests in 0.111s
FAILED (failures=1)
With readline-devel-6.0-4.el6.x86_64.

Updating the version check to 6.1 as in the patch from Martin certainly avoids the failing test.

Thanks, I can try to commit the version fix part of the patch when I get a chance. What is EL6.8, is that a Red Hat (Enterprise Linux) thing?

New changeset 55ec5fdc3099 by Martin Panter in branch '2.7':
Issue #19884: Avoid spurious output on OS X with Gnu Readline
https://hg.python.org/cpython/rev/55ec5fdc3099
New changeset 782d9b5d2e90 by Martin Panter in branch '3.5':
Issue #19884: Avoid spurious output on OS X with Gnu Readline
https://hg.python.org/cpython/rev/782d9b5d2e90
New changeset 72e034afeb55 by Martin Panter in branch 'default':
Issue #19884: Merge Readline updates from 3.5
https://hg.python.org/cpython/rev/72e034afeb55 

I committed my patch in full, so hopefully the Gnu Readline situation on OS X is also improved. Original fixes went into 3.4, but my patch only went into 3.5+.