homepage

subprocess.Popen() with universal_newlines=True does not convert line breaks correctly when the preferred encoding is UTF-16. For example, the following code--
code = r"import sys; sys.stdout.buffer.write('a\r\nb'.encode('utf-16'))"
args = [sys.executable, '-c', code]
popen = Popen(args, universal_newlines=True, stdin=PIPE, stdout=PIPE)
print(popen.communicate(input=''))
yields--
('a\n\nb', None)
instead of--
('a\nb', None)
The reason is that the code attempts to convert newlines before decoding to unicode instead of after:
http://hg.python.org/cpython/file/85266c6f9ae4/Lib/subprocess.py#l830
I am attaching a failing test case. I will upload a patch shortly.
Also see the related documentation issue 15561.

> when the preferred encoding is UTF-16
What is your operating system? How do you set the locale encoding to UTF-16? Do you mean UTF-16, UTF-16-LE or UTF-16-BE?

Attaching patch for review.

Victor, I have not personally experienced this issue. I just noticed that the order of operations is wrong or not portable in the _translate_newlines() method when I was looking at the code for another reason.

> How do you set the locale encoding to UTF-16? Do you mean UTF-16, UTF-16-LE or UTF-16-BE?
I confirmed that the issue occurs for all of these. For testing purposes, you can do--
locale.getpreferredencoding = lambda do_setlocale: 'utf-16'

There is a bug, but I'm not conviced that multibyte encodings are used as locale encoding.
About your patch: you should test the 3 types of newlines, so use a string like: '1\r\n2\r3\n4'.
+ # Popen() defaults to locale.getpreferredencoding(False).
+ locale.getpreferredencoding = lambda do_setlocale: 'utf-16'
FYI it's not directly Popen() which uses the locale encoding, but TextIOWrapper.

Thanks for your comments. 
> FYI it's not directly Popen() which uses the locale encoding, but TextIOWrapper.
Yes, I will note that to be more clear. Would you like me to add tests for UTF-16-LE, etc. (via a helper test method), or will UTF-16 alone suffice?

> Would you like me to add tests for UTF-16-LE, etc. (via a helper test method), or will UTF-16 alone suffice?
You can use a loop insteadd of an helper function.
I fail to see other encoding having the same issue, except the UTF-32 family (utf-32, utf-32-le, utf-32-be).
I don't think that writing a test for the 6 codecs is required, you can use utf-16 and utf-32-be. So you test without BOM and with BOM, and utf-16 and utf-32.

Updating patch with Victor's comments.

It looks like this also affects 3.2, but I will need to modify the test slightly because in 3.2, TextIOWrapper calls locale.getpreferredencoding() without any arguments.

Here is a patch also suitable for applying to Python 3.2.

Looks good for me, but please use assertEqual instead assertEquals.

Thank you, Andrew. Here is a patch updated with that change (and also merging with tip).

New changeset a0f7c2f79bce by Andrew Svetlov in branch '3.2':
Issue #15595: Fix subprocess.Popen(universal_newlines=True)
http://hg.python.org/cpython/rev/a0f7c2f79bce
New changeset aceb820154c3 by Andrew Svetlov in branch 'default':
Issue #15595: Fix subprocess.Popen(universal_newlines=True)
http://hg.python.org/cpython/rev/aceb820154c3 

Fixed. Thank you.