[Python-Dev] PEP 324: popen5 - New POSIX process module

Sat Jan 3 08:47:41 EST 2004

There's a new PEP available:
PEP 324: popen5 - New POSIX process module
A copy is included below. Comments are appreciated. 
----
PEP: 324
Title: popen5 - New POSIX process module
Version: $Revision: 1.4 $
Last-Modified: $Date: 2004年01月03日 10:32:53 $
Author: Peter Astrand <astrand at lysator.liu.se>
Status: Draft
Type: Standards Track (library)
Created: 19-Nov-2003
Content-Type: text/plain
Python-Version: 2.4
Abstract
 This PEP describes a new module for starting and communicating
 with processes on POSIX systems.
Motivation
 Starting new processes is a common task in any programming
 language, and very common in a high-level language like Python.
 Good support for this task is needed, because:
 - Inappropriate functions for starting processes could mean a
 security risk: If the program is started through the shell, and
 the arguments contain shell meta characters, the result can be
 disastrous. [1]
 - It makes Python an even better replacement language for
 over-complicated shell scripts.
 Currently, Python has a large number of different functions for
 process creation. This makes it hard for developers to choose.
 The popen5 modules provides the following enhancements over
 previous functions:
 - One "unified" module provides all functionality from previous
 functions.
 - Cross-process exceptions: Exceptions happening in the child
 before the new process has started to execute are re-raised in
 the parent. This means that it's easy to handle exec()
 failures, for example. With popen2, for example, it's
 impossible to detect if the execution failed.
 - A hook for executing custom code between fork and exec. This
 can be used for, for example, changing uid.
 - No implicit call of /bin/sh. This means that there is no need
 for escaping dangerous shell meta characters.
 - All combinations of file descriptor redirection is possible.
 For example, the "python-dialog" [2] needs to spawn a process
 and redirect stderr, but not stdout. This is not possible with
 current functions, without using temporary files.
 - With popen5, it's possible to control if all open file
 descriptors should be closed before the new program is
 executed.
 - Support for connecting several subprocesses (shell "pipe").
 - Universal newline support.
 - A communicate() method, which makes it easy to send stdin data
 and read stdout and stderr data, without risking deadlocks.
 Most people are aware of the flow control issues involved with
 child process communication, but not all have the patience or
 skills to write a fully correct and deadlock-free select loop.
 This means that many Python applications contain race
 conditions. A communicate() method in the standard library
 solves this problem.
Rationale
 The following points summarizes the design:
 - popen5 was based on popen2, which is tried-and-tested.
 - The factory functions in popen2 have been removed, because I
 consider the class constructor equally easy to work with.
 - popen2 contains several factory functions and classes for
 different combinations of redirection. popen5, however,
 contains one single class. Since popen5 supports 12 different
 combinations of redirection, providing a class or function for
 each of them would be cumbersome and not very intuitive. Even
 with popen2, this is a readability problem. For example, many
 people cannot tell the difference between popen2.popen2 and
 popen2.popen4 without using the documentation.
 - One small utility function is provided: popen5.run(). It aims
 to be an enhancement over os.system(), while still very easy to
 use:
 - It does not use the Standard C function system(), which has
 limitations.
 - It does not call the shell implicitly.
 - No need for quoting; using a variable argument list.
 - The return value is easier to work with.
 - The "preexec" functionality makes it possible to run arbitrary
 code between fork and exec. One might ask why there are special
 arguments for setting the environment and current directory, but
 not for, for example, setting the uid. The answer is:
 - Changing environment and working directory is considered
 fairly common.
 - Old functions like spawn() has support for an
 "env"-argument.
 - env and cwd are considered quite cross-platform: They make
 sense even on Windows.
 - No MS Windows support is available, currently. To be able to
 provide more functionality than what is already available from
 the popen2 module, help from C modules is required.
Specification
 This module defines one class called Popen:
 class Popen(args, bufsize=0, argv0=None,
 stdin=None, stdout=None, stderr=None,
 preexec_fn=None, preexec_args=(), close_fds=0,
 cwd=None, env=None, universal_newlines=0)
 Arguments are:
 - args should be a sequence of program arguments. The program to
 execute is normally the first item in the args sequence, but can
 be explicitly set by using the argv0 argument. The Popen class
 uses os.execvp() to execute the child program.
 - bufsize, if given, has the same meaning as the corresponding
 argument to the built-in open() function: 0 means unbuffered, 1
 means line buffered, any other positive value means use a buffer
 of (approximately) that size. A negative bufsize means to use
 the system default, which usually means fully buffered. The
 default value for bufsize is 0 (unbuffered).
 - stdin, stdout and stderr specify the executed programs' standard
 input, standard output and standard error file handles,
 respectively. Valid values are PIPE, an existing file
 descriptor (a positive integer), an existing file object, and
 None. PIPE indicates that a new pipe to the child should be
 created. With None, no redirection will occur; the child's file
 handles will be inherited from the parent. Additionally, stderr
 can be STDOUT, which indicates that the stderr data from the
 applications should be captured into the same file handle as for
 stdout.
 - If preexec_fn is set to a callable object, this object will be
 called in the child process just before the child is executed,
 with arguments preexec_args.
 - If close_fds is true, all file descriptors except 0, 1 and 2
 will be closed before the child process is executed.
 - If cwd is not None, the current directory will be changed to cwd
 before the child is executed.
 - If env is not None, it defines the environment variables for the
 new process.
 - If universal_newlines is true, the file objects fromchild and
 childerr are opened as a text files, but lines may be terminated
 by any of '\n', the Unix end-of-line convention, '\r', the
 Macintosh convention or '\r\n', the Windows convention. All of
 these external representations are seen as '\n' by the Python
 program. Note: This feature is only available if Python is
 built with universal newline support (the default). Also, the
 newlines attribute of the file objects fromchild, tochild and
 childerr are not updated by the communicate() method.
 The module also defines one shortcut function:
 run(*args):
 Run command with arguments. Wait for command to complete,
 then return the returncode attribute. Example:
 retcode = popen5.run("stty", "sane")
 Exceptions
 ----------
 Exceptions raised in the child process, before the new program has
 started to execute, will be re-raised in the parent. Additionally,
 the exception object will have one extra attribute called
 'child_traceback', which is a string containing traceback
 information from the child's point of view.
 The most common exception raised is OSError. This occurs, for
 example, when trying to execute a non-existent file. Applications
 should prepare for OSErrors.
 A PopenException will also be raised if Popen is called with
 invalid arguments.
 Security
 --------
 popen5 will never call /bin/sh implicitly. This means that all
 characters, including shell metacharacters, can safely be passed
 to child processes.
 Popen objects
 -------------
 Instances of the Popen class have the following methods:
 poll()
 Returns -1 if child process hasn't completed yet, or its exit
 status otherwise. See below for a description of how the exit
 status is encoded.
 wait()
 Waits for and returns the exit status of the child process.
 The exit status encodes both the return code of the process
 and information about whether it exited using the exit()
 system call or died due to a signal. Functions to help
 interpret the status code are defined in the os module (the
 W*() family of functions).
 communicate(input=None)
 Interact with process: Send data to stdin. Read data from
 stdout and stderr, until end-of-file is reached. Wait for
 process to terminate. The optional stdin argument should be a
 string to be sent to the child process, or None, if no data
 should be sent to the child.
 communicate() returns a tuple (stdout, stderr).
 Note: The data read is buffered in memory, so do not use this
 method if the data size is large or unlimited.
 The following attributes are also available:
 fromchild
 A file object that provides output from the child process.
 tochild
 A file object that provides input to the child process.
 childerr
 A file object that provides error output from the child
 process.
 pid
 The process ID of the child process.
 returncode
 The child return code. A None value indicates that the
 process hasn't terminated yet. A negative value means that
 the process was terminated by a signal with number
 -returncode.
Open Issues
 Perhaps the module should be called something like "process",
 instead of "popen5".
Reference Implementation
 A reference implementation is available from
 http://www.lysator.liu.se/~astrand/popen5/.
References
 [1] Secure Programming for Linux and Unix HOWTO, section 8.3.
 http://www.dwheeler.com/secure-programs/
 [2] Python Dialog
 http://pythondialog.sourceforge.net/
Copyright
 This document has been placed in the public domain.

Local Variables:
mode: indented-text
indent-tabs-mode: nil
sentence-end-double-space: t
fill-column: 70
End:
-- 
/Peter Åstrand <astrand at lysator.liu.se>