homepage

``python3.5 --help`` gives this information:
 -u : unbuffered binary stdout and stderr, stdin always buffered
However, stdout and stderr are actually always opened in text mode, and print() always expects a string and never a bytes object. This usage of "binary" in the --help is in contradiction with the usage of "binary" in the description of files (e.g. ``help(open)``).

Right. Moreover, "unbuffered" is wrong. It's line buffered: sys.stdout.buffer is directly a io.FileIO object, but TextIOWrapper only calls buffer.write() when the message contains a newline character. It's not (currently) possible to have a fully unbuffered stdout.

> It's not (currently) possible to have a fully unbuffered stdout.
Why doesn't create_stdio also pass `write_through = Py_True` when Py_UnbufferedStdioFlag is set? This would immediately pass writes through to the FileIO object, even without containing a newline (i.e. it sets text_needflush in _io_TextIOWrapper_write_impl).

I don't recall, I would have to search in old issues for the rationale. But
the explanation is probably performance, reduce the number of syscalls.

The write_through argument was added only in 3.3. I think this is a good idea to pass `write_through = Py_True` when Py_UnbufferedStdioFlag is set. Using -u means that performance is not important (that is why this is not default behavior).

Would it make sense to have two modes: line buferred and unbuffered,
as the C function setvbuf() for stdout?

The output of "python3.5 --help" says:
 -u : unbuffered binary stdout and stderr, stdin always buffered;
 also PYTHONUNBUFFERED=x
 see man page for details on internal buffering relating to '-u'
If you look at the man page as instructed then you'll see a clearer
explanation:
 -u Force the binary I/O layers of stdout and stderr to be
 unbuffered. stdin is always buffered. The text I/O layer will
 still be line-buffered.
For example, if you try this:
 python3.5 -uc 'import sys,time;w=sys.stdout.buffer.write;w(b"a");time.sleep(1);w(b"b");'
then you'll see that the binary output is indeed unbuffered as
documented.
The output of --help is trying to abbreviate this explanation, but I
think it's abbreviated too much. The explanation from the man page
seems clear to me, and is only a little longer, so I suggest changing
the --help output to match the man page.

Here's a patch that copies the text for the -u option from the man page to the --help output.

I just ran into this discrepancy working on the test cases for PEP 538 - +1 for Gareth's suggested approach of just aligning the `--help` output with the man page.

Issue30404 makes stdout and stderr truly unbuffered when run with -u.

Pull request for issue 30404 has been merged so we only need the documentation patch for the 3.6 branch (unfortunately 3.5 is now in security-fix-only mode) I've opened PR 3954.

New changeset 5f908005ce16b06d5af7b413264009c4b062f33c by Berker Peksag in branch '3.6':
bpo-28647: Update -u documentation (GH-3954)
https://github.com/python/cpython/commit/5f908005ce16b06d5af7b413264009c4b062f33c

Thank you for the patch, Gareth.

You're welcome.

I just checked the master branch:
-u : unbuffered binary stdout and stderr, stdin always buffered;
 also PYTHONUNBUFFERED=x
 see man page for details on internal buffering relating to '-u'
The doc is wrong. stdout and stderr are fully unbuferred since Serhiy changed them: commit 77732be801c18013cfbc86e27fcc50194ca22c8e, bpo-30404.

Good catch, I thought it was already fixed in master after bpo-30404. I've opened PR 3961.

Don't forget to update Misc/python.man.

The -u option doesn't affect the buffering of stdin. Is it worth to document this explicitly? Or it just adds a noise? Maybe remove this from the help and manpage but add in the RST documentation?

I think it's better to be explicit and mention 'stdin' in this case. Plus, "stdin is always buffered" is quite short so potential of creating noise is quite minimal IMO.
(FYI, I asked this question on GitHub: https://github.com/python/cpython/pull/3961#issuecomment-336035358)

In Python 2 there is an internal buffering in xreadlines(), readlines() and file-object iterators. You need to enter many lines first that the program get the first of them. And -u doesn't help.
But in Python 3 the program gets the input right as it becomes available. Reading is not blocked if the input is available. There are internal buffers, but they affect only performance, not the behavior. If you can edit a line before pressing Enter, this is because your terminal buffers a line before sending it to the program. I think it is more correct to say that stdin is always unbuffered in Python 3.

Serhiy: "(...) I think it is more correct to say that stdin is always unbuffered in Python 3."
I disagree. Technically, sys.stdin.read(1) reads up to 1024 bytes from the file descriptor 0. For me, "unbuffered read" means that read(1) reads a single byte.
Expected behaviour of an fully unbuffered stdin:
assert sys.stdin.read(1) == 'a'
assert os.read(0, 1) == b'b'
The program should not fail with an assertion error nor block if you write 'ab' characters into stdin.

Serhiy Storchaka:
https://github.com/python/cpython/pull/3961#issuecomment-336136160
"I suggest to continue the discussion on the tracker."
Ok, let's continue here.
"We are fixing the outdated documentation inherited from Python 2. First than keep some statement we should consider what it means in the context of Python 2 and what it means in the context of Python 3."
stdin buffering is a complex thing.
When running the UNIX command "producer | consumer", many users are confused by the buffering on the *producer* side.
When running a program in a TTY, the TTY does line buffering for you, you cannot get immediately a single character (without changing the default TTY configuration).
I don't think that we need to say too much. I just suggest to say "stdin is always buffered". That's all.
See my previous messages for the my definition of "buffered" versus "unbuffered" read.
Note: Today I learned the UNIX "stdbuf" command, useful to configure the stdin, stdout and stderr buffering of C applications using <stdio.h>.

stdin is mentioned in the documentation of the -u option only due to weird internal buffering in Python 2, because user can expect that -u disables it. It is documented what methods use internal buffering and how get rid of it. No other buffering is mentioned.
This no longer actual in Python 3. I think there is no need to mention stdin in the context of the -u option at all. -u doesn't affect stdin buffering, whatever that would mean. Period.
Alternatively you can include a lecture about different kinds of buffering and how -u doesn't affect them.

> -u doesn't affect stdin buffering, whatever that would mean.
I think we need to document behavior of stdin somewhere, because current the sys.stdin documentation states:
> When interactive, standard streams are line-buffered. Otherwise, they
> are block-buffered like regular text files. You can override this value
> with the -u command-line option.
https://docs.python.org/3/library/sys.html#sys.stdin 

Interesting comment in create_stdio() of Python/pylifecycle.c:
---
 /* stdin is always opened in buffered mode, first because it shouldn't
 make a difference in common use cases, second because TextIOWrapper
 depends on the presence of a read1() method which only exists on
 buffered streams.
 */
 if (Py_UnbufferedStdioFlag && write_mode)
 buffering = 0;
 else
 buffering = -1;
---
stdin is always buffered ;-)
I created bpo-31775: "Support unbuffered TextIOWrapper".

Serhiy: "I think there is no need to mention stdin in the context of the -u option at all. -u doesn't affect stdin buffering, whatever that would mean. Period."
Hum, I propose to mention stdin in -u documentation as: "The option has no effect on stdin." What do you think?

> I think we need to document behavior of stdin somewhere, because current the sys.stdin documentation states:
>
>> When interactive, standard streams are line-buffered. Otherwise, they
>> are block-buffered like regular text files. You can override this value
>> with the -u command-line option.
The last sentence is wrong and should be removed from sys.stdin documentation, no?

> Hum, I propose to mention stdin in -u documentation as: "The option has no effect on stdin." What do you think?
This LGTM too. More precise, it has no effect on stdin buffering. It has effect on the line_buffering attribute, but this attribute has no effect on reading.
> The last sentence is wrong and should be removed from sys.stdin documentation, no?
Or correct it, making it related only to stdout and stderr.

New changeset 7f580970836b0f6bc9c5db868d95bea81a3e1558 by Berker Peksag in branch 'master':
bpo-28647: Update -u documentation after bpo-30404 (GH-3961)
https://github.com/python/cpython/commit/7f580970836b0f6bc9c5db868d95bea81a3e1558

Thank you for reviews, Serhiy and Victor.

Thanks Berker for this nice documentation enhancement! It was required.
Do we need to update Python 3.6 documentation using the commit 5f908005ce16b06d5af7b413264009c4b062f33c, or are we good? (sorry, I didn't check)

Modules/main.c and Python.man is same in 3.6 branch. We could backport the change in Doc/library/sys.rst from 7f580970836b0f6bc9c5db868d95bea81a3e1558 but I didn't do it yet since it needs be manually backported.