homepage

The attached program uncovers a two-fold performance regression in
Python 2.5 with respect to Python 2.4. Below, the "element-wise" case
goes from 2.5 seconds in 2.4 to about 4 in 2.5. Since that's a pretty
serious increase, I thought I'd point it out.
$ python2.5 periodic_test.py
elwise 3.91989398003
collective 1.21577095985
$ python2.4 periodic_test.py
elwise 2.50014710426
tuplewise 1.35589385033 

I've narrowed the problem to the usage of generator expressions. 
Generator expressions appear to be MUCH slower on 2.5 than on 2.4.
>python -m timeit "tuple([1 for _ in xrange(3)])"
2.4 -> 2.23us
2.5 -> 2.31us (a bit slower, but not so much)
>python -m timeit "tuple(1 for _ in xrange(3))"
2.4 -> 3.32us
2.5 -> 8.03us (240% slower than 2.4)
It appears to be a fixed cost, or startup cost, of the generator 
expression; differences get smaller when building large tuples (but 
always Python 2.5 runs slower than 2.4)

May this be a byproduct of the new generator features in 2.5?

The slowdown is due to the new function _PyObject_LengthHint: on my
Win2K machine, 
the command
 python -m timeit "tuple(1 for _ in xrange(3))"
answers:
 100000 loops, best of 3: 4.14 usec per loop
By adding a line "return rv;" on the second line of
_PyObject_LengthHint, I get:
 100000 loops, best of 3: 2.71 usec per loop
Should we disable this function for some known built-in types?

I'll take a look at it next week -- want to think through the various 
cases. The current setup provides nice speedups when the length_hint 
is available. Possibly, it may be worthwhile to skip that check when 
the input is a generator. For the most part, I'm more concerned about 
the inner-loop time than the constant startup time outside the loop.

FWIW, the 2.4 to 2.5 timing difference came from renaming __len__ to 
__length_hint__. This was at Guido's request so that the value of bool
(iter(obj)) would always be True. The consequence of the change was 
that we lost the fast slot lookup for __len__ and instead have to do a 
dictionary based attribute lookup.
For the most part, I don't care about the overhead as it is constant. 
The inner-loop cost dominates. If you do care, the choices are to add 
some ugly, hackish specific type checks to bypass the attribute lookup 
in cases like generator objects where we know the type is absent. A 
more general, cleaner solution is to add a new slot for a length hint. 
Personally, I would rather leave this as is and live with the small 
constant lookup time. If you concur, please close this report. If 
not, please submit a patch for adding a new slot (model the code after 
that in PyObject_Size()).