[Python-checkins] r74700 - peps/trunk/pep-3145.txt

Mon Sep 7 07:18:06 CEST 2009

Author: brett.cannon
Date: Mon Sep 7 07:18:06 2009
New Revision: 74700
Log:
Add PEP 3145: Asynchronous I/O For subprocess.Popen
Added:
 peps/trunk/pep-3145.txt (contents, props changed)
Added: peps/trunk/pep-3145.txt
==============================================================================

--- (empty file)
+++ peps/trunk/pep-3145.txt	Mon Sep 7 07:18:06 2009
@@ -0,0 +1,135 @@
+PEP: 3145
+Title: Asynchronous I/O For subprocess.Popen
+Version: $Revision$
+Last-Modified: $Date$
+Author: (James) Eric Pruitt, Charles R. McCreary, Josiah Carlson
+Status: Draft
+Type: Standards Track
+Content-Type: text/plain
+Created: 04-Aug-2009
+Python-Version: 3.2
+Post-History:
+
+Abstract:
+
+ In its present form, the ``subprocess.Popen`` implementation is prone to
+ dead-locking and blocking of the parent Python script while waiting on data
+ from the child process. This PEP proposes to make
+ ``subprocess.Popen`` more asynchronous to help alleviate these
+ problems.
+
+Motivation:
+
+ A search for "python asynchronous subprocess" will turn up numerous
+ accounts of people wanting to execute a child process and communicate with
+ it from time to time reading only the data that is available instead of
+ blocking to wait for the program to produce data [1] [2] [3]. The current
+ behavior of the subprocess module is that when a user sends or receives
+ data via the stdin, stderr and stdout file objects, dead locks are common
+ and documented [4] [5]. While ``communicate`` can be used to alleviate some of
+ the buffering issues, it will still cause the parent process to block while
+ attempting to read data when none is available to be read from the child
+ process. 
+
+Rationale:
+
+ There is a documented need for asynchronous, non-blocking functionality in
+ ``subprocess.Popen`` [6] [7] [2] [3]. Inclusion of the code would improve the
+ utility of the Python standard library that can be used on Unix based and
+ Windows builds of Python. Practically every I/O object in Python has a
+ file-like wrapper of some sort. Sockets already act as such and for
+ strings there is ``StringIO``. Popen can be made to act like a file by simply
+ using the methods attached the the ``subprocess.Popen.stderr``,
+ ``stdout`` and
+ ``stdin`` file-like objects. But when using the read and write methods of
+ those options, you do not have the benefit of asynchronous I/O. In the
+ proposed solution the wrapper wraps the asynchronous methods to mimic a
+ file object.
+
+Reference Implementation:
+
+ I have been maintaining a Google Code repository that contains all of my
+ changes including tests and documentation [9] as well as blog detailing
+ the problems I have come across in the development process [10]. 
+
+ I have been working on implementing non-blocking asynchronous I/O in the
+ ``subprocess.Popen`` module as well as a wrapper class for
+ ``subprocess.Popen``
+ that makes it so that an executed process can take the place of a file by
+ duplicating all of the methods and attributes that file objects have. 
+
+ There are two base functions that have been added to the
+ ``subprocess.Popen``
+ class: ``Popen.send`` and ``Popen._recv``, each with two separate implementations,
+ one for Windows and one for Unix based systems. The Windows
+ implementation uses ``ctypes`` to access the functions needed to control pipes
+ in the kernel 32 DLL in an asynchronous manner. On Unix based systems,
+ the Python interface for file control serves the same purpose. The
+ different implementations of ``Popen.send`` and ``Popen._recv`` have identical
+ arguments to make code that uses these functions work across multiple
+ platforms. 
+
+ When calling the ``Popen._recv`` function, it requires the pipe name be
+ passed as an argument so there exists the ``Popen.recv`` function that passes
+ selects ``stdout`` as the pipe for ``Popen._recv`` by default.
+ ``Popen.recv_err``
+ selects ``stderr`` as the pipe by default. ``Popen.recv`` and
+ ``Popen.recv_err``
+ are much easier to read and understand than
+ ``Popen._recv('stdout' ...)`` and
+ ``Popen._recv('stderr' ...)`` respectively. 
+
+ Since the ``Popen._recv`` function does not wait on data to be produced
+ before returning a value, it may return empty bytes.
+ ``Popen.asyncread``
+ handles this issue by returning all data read over a given time
+ interval. 
+
+ The ``ProcessIOWrapper`` class uses the ``asyncread`` and
+ ``asyncwrite`` functions to
+ allow a process to act like a file so that there are no blocking issues
+ that can arise from using the ``stdout`` and ``stdin`` file objects produced from
+ a ``subprocess.Popen`` call.
+ 
+
+References:
+
+ [1] [ python-Feature Requests-1191964 ] asynchronous Subprocess
+ http://mail.python.org/pipermail/python-bugs-list/2006-December/
+ 036524.html
+
+ [2] Daily Life in an Ivory Basement : /feb-07/problems-with-subprocess
+ http://ivory.idyll.org/blog/feb-07/problems-with-subprocess
+
+ [3] How can I run an external command asynchronously from Python? - Stack 
+ Overflow
+ http://stackoverflow.com/questions/636561/how-can-i-run-an-external-
+ command-asynchronously-from-python
+
+ [4] 18.1. subprocess - Subprocess management - Python v2.6.2 documentation
+ http://docs.python.org/library/subprocess.html#subprocess.Popen.wait
+
+ [5] 18.1. subprocess - Subprocess management - Python v2.6.2 documentation
+ http://docs.python.org/library/subprocess.html#subprocess.Popen.kill
+
+ [6] Issue 1191964: asynchronous Subprocess - Python tracker
+ http://bugs.python.org/issue1191964
+
+ [7] Module to allow Asynchronous subprocess use on Windows and Posix
+ platforms - ActiveState Code
+ http://code.activestate.com/recipes/440554/
+
+ [8] subprocess.rst - subprocdev - Project Hosting on Google Code
+ http://code.google.com/p/subprocdev/source/browse/doc/subprocess.rst?spec=svn2c925e935cad0166d5da85e37c742d8e7f609de5&r=2c925e935cad0166d5da85e37c742d8e7f609de5#437
+
+ [9] subprocdev - Project Hosting on Google Code
+ http://code.google.com/p/subprocdev
+
+ [10] Python Subprocess Dev
+ http://subdev.blogspot.com/
+
+Copyright:
+
+ This P.E.P. is licensed under the Open Publication License;
+ http://www.opencontent.org/openpub/.
+
