homepage

In 3.2.2 the newline for print() is \n on Windows, and not \r\n as expected.
In 3.1.4 the newline is \r\n.
OS is Win 7, and tried on both 32 and 64 bit.
Small example with output is attached.

To people who open the file in their browser: text files are very similar, but newline_3.1.txt has CRLF line endings and newline_3.2.txt has LF line endings.
M.Z, how did you obtain them? did you start a subprocess?

Hi Amaury,
The two text files were obtained through redirection in Windows, so I simply ran the newline.py file with:
 ...> C:\Python31\python.exe newline.py > newline_3.1.txt
 ...> C:\Python32\python.exe newline.py > newline_3.2.txt
Best regards,
Morten Zilmer

I changed how newlines are handled on Windows to fix an issue with CGI: see the issue #10841.
changeset: 67431:0933c3753a71
user: Victor Stinner <victor.stinner@haypocalc.com>
date: Fri Jan 07 18:47:22 2011 +0000
files: Misc/NEWS Modules/_io/fileio.c Modules/main.c Parser/tokenizer.c
description:
Issue #10841: set binary mode on files; the parser translates newlines
On Windows, set the binary mode on stdin, stdout, stderr and all
io.FileIO objects (to not translate newlines, \r\n <=> \n). The Python parser
translates newlines (\r\n => \n).

print() uses PyFile_WriteString("\n", file) by default (if the end argument is not set) to write the newline. TextIOWrapper.write("\n") replaces "\n" by TextIOWrapper._writenl.
On Windows, stdin, stdout and stderr are creates using TextIOWrapper(..., newline=None). In this case, TextIOWrapper._writenl is os.linesep and so '\r\n'.
To sum up, print() writes '\n' into sys.stdout, but sys.stdout write b'\r\n' into the file descriptor 1 which is a binary file (ie. the underlying OS file doesn't translate newlines).
If the output is redirected (e.g. into a file), TextIOWrapper is created with line_buffering=False.
You may try to force line_buffering=True when the output is redirected.

> If the output is redirected (e.g. into a file),
> TextIOWrapper is created with line_buffering=False.
How does this affect the \r\n translation?

Just to make it clear: I have not observed any problems on the Windows terminal (cmd) with \n newline, but at least Notepad does not break lines correctly if only \n is used.

I found 'more' command in Windows7 requires \r\n.
Python 2.7.3:
C:\>python -c "for i in range(5):print(i)"|more
0
1
2
3
4
Python 3.3(trunk):
c:\src\cpython\PCbuild>python -c "for i in range(5):print(i)"|more
?????

> On Windows, stdin, stdout and stderr are creates using TextIOWrapper(..., newline=None).
> In this case, TextIOWrapper._writenl is os.linesep and so '\r\n'.
Oh, I was wrong: stdin is created with newline=None, but stdout and stderr are created with newline="\n" and so "\n" is not translated to "\r\n".
I checked in Python 2.7: print("abc") and sys.stdout.write("abc\n") writes b"abc\r\n" into the output file (when the output is redirected), but sys.stdout.write("abc\r\n") writes b"abc\r\r\n". Python 3.3 should do the same: \r\n is preferred on Windows (ex: notepad doesn't support UNIX line ending, \n).
Attached patch changes line ending for stdout and stderr on Windows: translate "\n" to "\r\n".
It would be nice to fix this before Python 3.3 final.

Test for this issue. Tested on Windows7, Ubuntu linux 12.04.
I wonder why "print(1, file=sys.stderr)" returns '1' instead of '1\n'.
But in Python2.7, "print >>sys.stderr, 1" also returns '1', 
so this might not be a problem.

> I wonder why "print(1, file=sys.stderr)" returns '1' instead of '1\n'.
I suppose that you mean "returns '1\n' instead of '1'". This is a
major change between Python 2 and Python 3. Use print(1, end=' ') if
you want the same behaviour. See:
http://docs.python.org/dev/whatsnew/3.0.html#print-is-a-function
You can also use the print() as a function in Python 2 using "from
__future__ import print_function":
http://docs.python.org/dev/whatsnew/2.6.html#pep-3105-print-as-a-function
I never liked "print expr," because it looks like a typo or an ugly
hack. It's easy to add a comma by mistake.

> It would be nice to fix this before Python 3.3 final.
@Georg: So, what do you think?

About the patch: why wouldn't you use newline = NULL in both cases?

On Fri, Aug 3, 2012 at 5:16 AM, STINNER Victor <report@bugs.python.org> wrote:
>> I wonder why "print(1, file=sys.stderr)" returns '1' instead of '1\n'.
>
> I suppose that you mean "returns '1\n' instead of '1'". 
No, sorry for my lame wording.
In the test I submitted, printing to stdout with 
 
 "print(1, file=sys.stdout);print(2, file=sys.stdout)"
outputs
 "1\r\n2\r\n"
but printing to stderr with 
 "print(1, file=sys.stderr);print(2, file=sys.stderr)" 
outputs
 "1\r\n2" <- no '\r\n' at the end
I wondered why, but this is not specific to Python 3. 
With Python 2.7 
 print >>sys.stderr, 1
doesn't output '\r\n' at the end also. So I think this may 
not be a bug.

New changeset c55dbb84f3b4 by Victor Stinner in branch 'default':
Close #13119: use "\r\n" newline for sys.stdout/err on Windows
http://hg.python.org/cpython/rev/c55dbb84f3b4 

New changeset 09408b990ca5 by Victor Stinner in branch '3.2':
Close #13119: use "\r\n" newline for sys.stdout/err on Windows
http://hg.python.org/cpython/rev/09408b990ca5 

Windows buildbots now show failures in the test suite.

New changeset 4efad7fba42a by Antoine Pitrou in branch '3.2':
Fix test_sys under Windows (issue #13119)
http://hg.python.org/cpython/rev/4efad7fba42a
New changeset e4a87f0253e9 by Antoine Pitrou in branch 'default':
Merge universal newlines-related fixes (issue #13119)
http://hg.python.org/cpython/rev/e4a87f0253e9 

New changeset f8e435d6a801 by Antoine Pitrou in branch 'default':
Fix test_venv to work with universal newlines (issue #13119)
http://hg.python.org/cpython/rev/f8e435d6a801 

test_httpservers still fails, it's the CGI tests...

Fix for test_httpservers

Sorry, please ignore the patch 'issue13119_httpserver.patch' I posted above. 
Behavior of "-u" commandline option in Python3.3 is differ than in Python 2. 
We should not convert newline characters if "-u" specified? I'll investigate more.

We should not convert \n with -u command line option or PYTHONUNBUFFERED was set.
Added a patch to fix error in test_httpservers.

> We should not convert \n with -u command line option or PYTHONUNBUFFERED was set.
Why that? What do universal newlines have to do with buffering?

I wonder why this is a release blocker. It's a bug in Python 3.2, so why should it block the release of 3.3 (it's not a regression).
If no complete solution is coming up, I recommend to revert all changes on this issue, and reconsider after the 3.3 release.

> I wonder why this is a release blocker. It's a bug in Python 3.2, so
> why should it block the release of 3.3 (it's not a regression).
It's a blocker because the fix broke a couple of tests.
And it's also a regression from 3.1 and 2.7.

> It's a blocker because the fix broke a couple of tests.
That cannot possibly be the explanation why haypo declared
it a blocker on 2012年08月01日; the fix was only applied on
2012年08月04日.
But I agree that it should block the release now; hence I
propose to roll back the entire set of changes and revert
to the 3.2 state.
> And it's also a regression from 3.1 and 2.7.
Since 3.2 was released with this behavior, this bug cannot manage
to block 3.3.

Antoine Pitrou added the comment:
>
>> We should not convert \n with -u command line option or PYTHONUNBUFFERED was set.
>
> Why that? What do universal newlines have to do with buffering?
Man page of Python says 
 -u Force stdin, stdout and stderr to be totally unbuffered. On
 systems where it matters, also put stdin, stdout and stderr in
 binary mode. 
test_httpservers depends on this behavior, but was implemented as documented in Python 3.

> Man page of Python says 
> 
> -u Force stdin, stdout and stderr to be totally unbuffered. On
> systems where it matters, also put stdin, stdout and stderr in
> binary mode. 
I don't know which version it is, but current 3.3 says:
"Force the binary I/O layers of stdin, stdout and stderr to be
unbuffered. The text I/O layer will still be line-buffered."
> test_httpservers depends on this behavior, but was implemented as
> documented in Python 3.
I would argue that test_httpservers is wrong, since it uses print() in a
CGI script where sys.stdout.buffer.write() should really be used.

New changeset bc4fdb758b8c by Antoine Pitrou in branch '3.2':
Fix CGI tests to take into account the platform's line ending (issue #13119)
http://hg.python.org/cpython/rev/bc4fdb758b8c
New changeset ee185c6b2880 by Antoine Pitrou in branch 'default':
Fix CGI tests to take into account the platform's line ending (issue #13119)
http://hg.python.org/cpython/rev/ee185c6b2880 

Buildbots are happy, the issue can be closed again. Thanks Antoine.

> I don't know which version it is, but current 3.3 says:
Ah, sorry, I thought I was reading latest Man page.

> + " newline is '' or '\n', no translation takes place. If newline is any\n"
Non-escaped "\n".

Would be nice to be a bit more specific *where* that line comes from.

> Would be nice to be a bit more specific *where* that line comes from.
Modules/_io/textio.c, changesets 243ad1a6f638 and 083776adcacc.

Oops, I got the wrong issue, sorry.