Expose Python standard IO to Subprocess

Question

On Python 3.5.1, I have the following:

output = subprocess.check_output(cmd).decode(encoding="UTF-8")

This calls the properly invoked command cmd. C++14 code in cmd looks like:

HANDLE handle = GetStdHandle(STD_OUTPUT_HANDLE);
assert(handle!=INVALID_HANDLE_VALUE); //Always passes
assert(handle!=nullptr); //Always passes
CONSOLE_SCREEN_BUFFER_INFO csbi;
BOOL result = GetConsoleScreenBufferInfo(handle,&csbi);
assert(result!=0); //Always fails. `GetLastError()` returns 6 (invalid handle) 

Running the above Python code causes the subprocess cmd to fail at the indicated line. According to the Python docs, in this case, the stdout/stderr should be inherited from the parent process (i.e., the Python interpreter). So, it shouldn't. In fact, the above works just fine for e.g. printfed output.

Attempting to redirect explicitly also fails:

#Traceback (most recent call last):
# File "C:\dev\Python35\lib\subprocess.py", line 914, in __init__
# errread, errwrite) = self._get_handles(stdin, stdout, stderr)
# File "C:\dev\Python35\lib\subprocess.py", line 1145, in _get_handles
# c2pwrite = msvcrt.get_osfhandle(stdout.fileno())
#io.UnsupportedOperation: fileno
p = subprocess.Popen(cmd.split(" "),stdout=sys.stdout,stderr=sys.stderr)
#Using `subprocess.PIPE` instead fails in the same way as the `subprocess.check_output(...)`
#example originally given above.

What's going wrong? How do I fix it?

• Eryk Sun: check_output sets stdout=PIPE. You can't use that handle with console functions such as GetConsoleScreenBufferInfo and WriteConsole. printf works because ultimately it calls WriteFile, which writes bytes to any kind of file handle.
• Eryk Sun: Is sys.stdout a filelike object in an IDE? In that case it's not a operating system file with a handle that can be inherited in the standard handles of the child process. If your C++ code falls back to using C standard I/O (preferably line unbuffered) when stdout isn't a console, then your Python code can use a thread to read from the p.stdout pipe. check_output and Popen.communicate do this for you, but you may need something more interactive.
• Eryk Sun: @HarryJohnston, we know that sys.stdout isn't associated with a Windows file (console, pipe, disk) because it doesn't have a fileno(). I think imallett wants standard output from the child process to show up on Python's sys.stdout, regardless of what sys.stdout is (e.g. sys.stdout could be a proxy object that talks to an IDE's interactive window using a pipe or socket). In that case a thread can loop over p.stdout.read(1) or p.stdout.readline() and write to sys.stdout.
• Harry Johnston: @eryksun: right you are. The call to .Popen should have worked as shown if the OP were talking about the simple scenario I was thinking of. At least, it works for me. :-)

Answer 1

A pipe is not a console. check_output() uses stdout=PIPE internally. It won't redirect the console output (produced by WriteConsoleW()).

The first error (invalid handle) suggests that the standard output is not the console device.

Unless it is a desired behavior (to print outside of the standard output); use WriteFile() if stdout is not a console device.

The second error (io.UnsupportedOperation: fileno) suggests that sys.stdout is not a real file and therefore you can't pass it as stdout parameter to a subprocess (you could redirect subprocess' stdout using stdout=PIPE and print the output using print() or sys.stdout.write() method directly).

• geometrian: After a bit of outside reading in addition to this answer, I think what I really wanted was for cmd to use the console opened by the Python interpreter. Since cmd is a subprocess, I'm not sure that desire makes sense.¶ When invoking cmd, however, cmd seems to open its own console window (a new one appears when run). Can you please instead explain why cmd doesn't recognize this window and write to it when, for example, there is no redirection in the Python invocation?
• jfs: @imallett: "How a C++ application can use the console opened by its parent Python script (via subprocess)" sounds like a good question but it would be the third issue (try to limit your questions to a single issue per question). Have you tried the obvious? (don't redirect stdout, just don't pass stdout parameter e.g., use subprocess.check_call(cmd)—the subprocess should print to the console in this case). Also, it seems your code assumes that the stdout is the console (I initially missed it—I've updated the answer accordingly).
• geometrian: I didn't mean to ask how I could use the parent's console, as that would indeed be a separate question. I meant I was unclear on how what I did failed.¶ Attempting the suggested subprocess.check_call(cmd) works (the assertions pass, and the output is printed to the console). When redirected, it crashes.¶ I think I understand: the handle is generated from the stdout handle. When redirected, that's a pipe, not a file directed into a terminal. So the terminal still exists, but since stdout is not attached to it, getting the handle returns the pipe instead, So terminal-ish things fail. Yes?
• jfs: @imallett yes. The gist is correct: "the handle is generated from the stdout handle. When redirected, that's a pipe, not a file directed into a terminal."
• Eryk Sun: @imallett, if you know you're attached to a console (e.g. GetConsoleCP returns a non-zero value), you can open \\.\CONOUT$ to write to it. It should be opened with "r+" access.