homepage

From a thread on python-dev...
http://mail.python.org/pipermail/python-dev/2008-January/076446.html
Mike Kent mike.kent at sage.com
Thu Jan 24 16:33:47 CET 2008
Recently I was trying to debug an old python program who's maintenance I
inherited. I was using the quick-and-dirty method of putting some 'print
>>sys.stderr' statements in the code, and then running the command with
'2>filename' appended to the end of the command line. Imagine my surprise
to see that all of the prompt text from the program's raw_input calls were
also disappearing from the screen output, and appearing in the stderr
output routed to the file.
The latest documentation for raw_input states "If the prompt argument is
present, it is written to standard output without a trailing newline."
I posted a question regarding the observed behavior to comp.lang.python
and Gabriel Genellina (thanks Gabriel!) pointed out that despite the
documentation, raw_input was hard-coded to always output its prompt text
to stderr.
This raises two questions:
1. Shouldn't the current documentation be corrected to state that raw_input
writes its prompt to standard error?
2. Is this really the hard-coded behavior we want? I don't think my
use-case is that odd; in fact, what I find very odd is that the prompt
output is send to stderr. I mean, I'm printing the prompt for a question,
not some error message. Can there not at least be an optional parameter to
indicate that you want the output sent to stdout rather than stderr?
... after a few responses ...
Guido van Rossum guido at python.org
Thu Jan 24 21:09:12 CET 2008
On Jan 24, 2008 11:41 AM, Mike Kent <mike.kent at sage.com> wrote:
...
> Interesting point about whether GNU readline is installed. My setup
is RedHat
> Linux, with Python 2.5 that I built and installed myself. GNU
readline is not,
> in fact, installed. If you look at Python2.5/Parser/myreadline.c,
function
> PyOS_StdioReadline, line 125, you will see that prompt output is being
sent to
> stderr. As best as my Python-fu can determine, this is the code used
to output
> a raw_input prompt (thanks again to Gabriel Genellina for pointing me
in the
> right direction.)
>
> It's entirely likely that the difference in what I am seeing and what
you guys
> are seeing is caused by my not having GNU readline installed. 
Nevertheless,
> the behavior without it seems wrong, and is certainly different from the
> documentation.
Agreed.

GNU readline is configured as to prompt the user using standard output, 
and read input from standard input; if this is the desired behavior it 
would be easy to provide a simple patch so input/raw_input behave that 
way even when readline is not used.

Any *NIX gurus who can sort this one?

From reading the code for raw_input in 2.7 or input in 3.3 (Python/bltinmodule.c:1573), it looks to me that stdout is used, which would mean this issue is fixed. However I browsed the file history and could not find the commit that changed this, and my C skills are limited, so I’m adding Ezio to nosy to have another pair of eyes confirm.

The code that dictates this behavior is in /Parser/myreadline.c and has not been rectified yet in either Python 2.7 (http://hg.python.org/cpython/file/bfdf366a779a/Parser/myreadline.c#l107) or the default branch (http://hg.python.org/cpython/file/c64dec45d46f/Parser/myreadline.c#l111). Specifically, within these functions, references to standard error should actually be references to standard out.
The attached file is a proposed patch for this bug on the 2.7 branch, bringing interpreter behavior into accordance with the Python documentation (http://docs.python.org/library/functions.html#raw_input), which states that the prompt is written to standard out, as opposed to standard error.

Also uploading a patch for the Python3.2 branch.

Please see this stackoverflow thread where more information is given about this issue:
http://stackoverflow.com/questions/14009714/strange-redirection-effect-with-raw-input 

I experimented with various redirections to /dev/null, files, and other terminal windows on Linux. Current behaviour I am seeing seems to be something like this:
* Prefers prompting to stderr if both stdout and stderr are terminals
* Prefers prompting to stdout if neither are terminals
* Prompts to the non-terminal if only one of stderr and stdout is a terminal. Surely this one should be the other way around if there is going to be a preference at all?

For the record, this bug is still open. 
The proposed patch is not merged in any of branches.
The prompt for raw_input in all versions, go to stderr.

See also issue #24402: input() uses sys.__stdout__ instead of sys.stdout for prompt

The input() implementation is a bit like this:
def input(prompt):
 if stdin and stdout are the original file descriptors, and are terminals:
 return PyOS_Readline(sys.stdin, sys.stdout, prompt)
 else:
 sys.stdout.write(prompt) # Writes to stdout
 return sys.stdin.readline()
def PyOS_StdioReadline(stdin, stdout, prompt):
 '''Default implementation of PyOS_Readline()'''
 sys.stderr.write(prompt) # Writes to stderr
 return stdin.readline()
def call_readline(stdin, stdout, prompt):
 '''Implementation of PyOS_Readline() in the "readline" module'''
 rl_instream = stdin
 rl_outstream = stdout # Readline writes to stdout
 return readline_until_enter_or_signal(prompt)
It looks like PyOS_StdioReadline() has always written to stderr. The stdin and stdout parameters of PyOS_Readline() were added later, in revision dc4a0336a2a3.
I think the changes to myreadline.c will also affect the interactive interpreter prompt. But we have Issue 12869 open to change that to stdout, so maybe the change is okay.
Since input() should no longer depend on any instance of stderr, perhaps the check for "lost sys.stderr" should also be removed.
It may be worth applying any changes in myreadline.c to the independent version in pgenmain.c as well, just for consistency.

+1 to applying this patch. After reviewing this and Issue 12869, I don't see any substantial objections or concerns. The status is "test needed". Is a test really needed? My instinct that simply aligning the implementation with the docs is sufficient.

"Test needed" is meant to mean someone needs help producing the problem, but people also seem use it to request a refined test case for the test suite.
A test case is always nice, although in this case it is a bit tricky. I can try to knock one up use the existing infrastructure in test_builtin.PtyTests. A similar test case could probably be made for the interactive interpreter (Issue 12869), but might be more involved, and I don’t think there is any existing code to copy from.

Here is an updated patch for Python 3.
I did not remove the "lost sys.stderr" check I mentioned earlier, because the implementation still needs it to call flush().
Changes compared to Michael’s patch:
* Added a test for input() using a pseudoterminal and subprocess.Popen
* Write to the passed-in sys_stdout parameter, not the global stdout
* Continue to call fflush(stderr), to avoid regressions with buffered stderr messages
* Updated /Parser/pgenmain.c to use sys_stdout
I also had to update test_cmd_line_script, which expected the prompt to be on stderr. This made me wonder if it is a good idea to change where the interpreter prompt (>>>) goes in a bugfix release. AFAIK it is not documented, and it could potentially break other things that use the interactive interpreter. What do people think? A way to avoid this might be to pass stderr as the sys_stdout parameter.
Also, it would be awesome if someone could try my new test_builtins test case on BSD or OS X. I only tested it with Linux. The last time I messed with pseudoterminals like this I caused the tests to hang on BSD buildbots.

The entire test suite passes with the v2 patch on my OSX 10.10.

Tal: thanks for testing.
This v3 patch changes the interactive interpreter to pass stderr as the sys_stdout parameter. This means we should maintain compatibility with the interpreter prompt going to stderr, but still fix input().

Unix shell builtin command "read" outputs prompt to stderr. In bash that uses readline and in dash that doesn't use readline. Looks as this is standard behavior.

The way I see it, input() is mainly geared for prompting to stdout, and it is just one aspect that strangely uses stderr:
* Documentation says stdout
* Stdout is checked if it is a terminal and not redirected
* Gnu Readline is configured for stdout
* The fallback for non-terminals uses stdout
Arguments for using stderr:
* Consistency with Unix shell
* Consistency with the Python interactive interpreter prompt
Maybe it is more ideal to use stderr (I have no idea). But I think that would be a more drastic change.

Serhiy, was your comment an objection to changing away from stderr, or was that just an observation that Python’s design is inconsistent with the rest of the world?

This is rather an objection.
If Gnu Readline is configured for stdout, why bash outputs to stderr? We should investigate what exactly do bash and other popular programs with readline, and implement this in Python.
Changing the documentation usually is a less drastic change than changing behavior.

Okay, I see. To clarify, it is Python that sets up Gnu Readline for stdout: <https://hg.python.org/cpython/file/v3.5.1/Python/bltinmodule.c#l1941>. The problem is whichever way we go, we will have to change some part of the behaviour to make it internally consistent. I think my patch is the minimal change required.

I think we first should fix 3.6 in correct way, and then see what we can backport to other branches without breaking too much. Or left them as is. For example the output of --version was changed from strerr to stdout (issue18338, issue18920) in default branch, but this was not backported.

This proposal is starting to sound reasonable to me. Changing the title to reflect the new direction.

Regarding the comment by Martin Panter from 2015年11月22日: It would be nice if PyOS_StdioReadline worked that way. Unfortunately, it's still based on C file objects and char* for the prompt string rather than using actual Python objects. The relevant issue is issue17620 .

Further discussion at https://discuss.python.org/t/builtin-function-input-writes-its-prompt-to-sys-stderr-and-not-to-sys-stdout/12955 

See also Issue31603.