[Python-Dev] accumulator display syntax

Wed Oct 22 01:19:58 EDT 2003

[Raymond]
> > If there is any doubt on that score, I would be happy to update
> > the PEP to match the current proposal for iterator expressions
> > and solicit more community feedback.

[Guido]
> Wonderful! Rename PEP 289 to "generator expressions" and change the
> contents to match this proposal. Thanks for being the fall guy!

Here is a rough draft on the resurrected PEP.
I'm sure it contains many flaws and I welcome suggested amendments.
In particular, the follow needs attention:
* Precise specification of the syntax including the edge cases
 with commas where enclosing parentheses are required.
* Making sure the acknowledgements are correct and complete.
* Verifying my understanding of the issues surrounding late binding,
 modification of locals, and returning generator expressions.
* Clear articulation of the expected benefits. There are so many,
 it was difficult to keep it focused.
Raymond Hettinger
----------------------------------------------------------------------
PEP: 289
Title: Generator Expressions
Version: $Revision: 1.2 $
Last-Modified: $Date: 2003年08月30日 23:57:36 $
Author: python at rcn.com (Raymond D. Hettinger)
Status: Active
Type: Standards Track
Created: 30-Jan-2002
Python-Version: 2.3
Post-History: 22-Oct-2003
Abstract
 This PEP introduces generator expressions as a high performance,
 memory efficient generalization of list expressions and
 generators.
Rationale
 Experience with list expressions has shown their wide-spread
 utility throughout Python. However, many of the use cases do
 not need to have a full list created in memory. Instead, they
 only need to iterate over the elements one at a time.
 For instance, the following dictionary constructor code will
 build a full item list in memory, iterate over that item list,
 and, when the reference is no longer needed, delete the list:
 d = dict([(k, func(v)) for k in keylist])
 Time, clarity, and memory are conserved by using an generator
 expession instead:
 d = dict((k, func(v)) for k in keylist)
 Similar benefits are conferred on the constructors for other
 container objects:
 s = Set(word for line in page for word in line.split())
 Having a syntax similar to list comprehensions makes it easy to
switch
 to an iterator expression when scaling up application.
 Generator expressions are especially useful in functions that reduce
 an iterable input to a single value:
 sum(len(line) for line.strip() in file if len(line)>5)
 Accordingly, generator expressions are expected to partially
eliminate
 the need for reduce() which is notorious for its lack of clarity.
And,
 there are additional speed and clarity benefits from writing
expressions
 directly instead of using lambda.
 List expressions greatly reduced the need for filter() and map().
 Likewise, generator expressions are expected to minimize the need
 for itertools.ifilter() and itertools.imap(). In contrast, the
 utility of other itertools will be enhanced by generator
expressions:
 dotproduct = sum(x*y for x,y in itertools.izip(x_vector,
y_vector))
BDFL Pronouncements
 The previous version of this PEP was REJECTED. The bracketed
 yield syntax left something to be desired; the performance gains had
 not been demonstrated; and the range of use cases had not been
 shown. After, much discussion on the python-dev list, the PEP has
 been resurrected its present form. The impetus for the discussion
 was an innovative proposal from Peter Norvig.
The Gory Details
 1) In order to achieve a performance gain, generator expressions
need
 to be run in the local stackframe; otherwise, the improvement in
 cache performance gets offset by the time spent switching
 stackframes. The upshot of this is that generator expressions need
 to be both created and consumed within the context of a single
 stackframe. Accordingly, the generator expression cannot be
returned
 to another function:
 return (k, func(v)) for k in keylist
 2) The loop variable is not exposed to the surrounding function.
 This both facilates the implementation and makes typical use cases
 more reliable. In some future version of Python, list
comprehensions
 will also hide the induction variable from the surrounding code
(and,
 in Py2.4, warnings will be issued for code accessing the induction
 variable).

 3) Variables references in the generator expressions will exhibit
late
 binding just like other Python code. In the following example, the
 iterator runs *after* the value of y is set to one:
 def h():
 y = 0
 l = [1,2]
 def gen(S):
 for x in S:
 yield x+y
 it = gen(l)
 y = 1
 for v in it:
 print v
 4) List comprehensions will remain unchanged.
 So, [x for x in S] is a list comprehension and
 [(x for x in S)] is a list containing one generator expression.
 5) It is prohibited to use locals() for other than read-only use
 in generator expressions. This simplifies the implementation and
 precludes a certain class of obfuscated code.
Acknowledgements:
 Peter Norvig resurrected the discussion proposal for "accumulation
 displays".
 Alex Martelli provided critical measurements that proved the
 the performance benefits of generator expressions.
 Samuele Pedroni provided the example of late binding.
 Guido van Rossum suggested the bracket free, yield free syntax.
 Raymond Hettinger first proposed "generator comprehensions" in
 January 2002.
References
 [1] PEP 255 Simple Generators
 http://python.sourceforge.net/peps/pep-0255.html
 [2] PEP 202 List Comprehensions
 http://python.sourceforge.net/peps/pep-0202.html
 [3] Peter Norvig's Accumulation Display Proposal
 http:///www.norvig.com/pyacc.html
Copyright
 This document has been placed in the public domain.
Local Variables:
mode: indented-text
indent-tabs-mode: nil
fill-column: 70
End: