[Python-Dev] Mailbox module - timings and functionality changes

Tue Jun 29 17:40:50 CEST 2010

Guido van Rossum wrote:
> On Tue, Jun 29, 2010 at 7:49 AM, Steve Holden <steve at holdenweb.com> wrote:
>> Steve Holden wrote:
>>> Nick Coghlan wrote:
>>>> Command line: ./python -m test.regrtest -v test_mailbox
>>>>>>>> trunk: Ran 274 tests in 25.239s
>>>> py3k: Ran 268 tests in 26.263s
>>>>>>>> So I don't see any substantial difference on a Kubuntu 10.04 box (both
>>>> builds are recent'ish, but not completely up to date).
>>>>>>>> However, the underlying IO access is significantly different between
>>>> POSIX and Windows, so there could still be something pathological
>>>> happening at the filesystem manipulation layer. My comparisons are
>>>> also 2.7 vs 3.2 rather than 2.6 vs 3.1.
>>>>>>>> Cheers,
>>>> Nick.
>>>>>>> Thanks for all the timings! If a Windows user could do the same thing
>>> that would help ...
>>>>> And there is *definitely a performance issue. I created a Thunderbird
>> folder of 26 Google alerts and just parsed then all after reading them
>> in from the mailbox.
>>>> 2.5 (!): 0.78 sec
>> 3.1 : 42.80 sec
>>>> Rather than debate the code here perhaps I should just open an issue for
>> this? I can then provide both a program and some data, which can be
>> added to the tests if appropriate. The issue can clearly stand some
>> investigation.
>> Since you have such a great reproducible test case, could you point
> the profiler at it? (Perhaps on a reduced dataset... The profiler
> multiples your run time by some number between 2 and 10 IIRC.)
>Sure. I attach the outputs of both files, as well as the program and the
data. With profiling (python -m cProfile test3.py) the run took less
than a third of a second under 2.5, and 168 seconds under 3.1. I'd say
that was problematical :)
I will leave the profiler output to speak for itself, since I can find
nothing much to say about it except that there's a hell of a lot of
decoding going on inside mailbox.iterkeys().
regards
 Steve
-- 
Steve Holden +1 571 484 6266 +1 800 494 3119
See Python Video! http://python.mirocommunity.org/
Holden Web LLC http://www.holdenweb.com/
UPCOMING EVENTS: http://holdenweb.eventbrite.com/
"All I want for my birthday is another birthday" -
 Ian Dury, 1942-2000
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: test3.1.out
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100629/53064c86/attachment-0004.ksh>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: test2.5.out
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100629/53064c86/attachment-0005.ksh>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: test3.py
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100629/53064c86/attachment-0006.ksh>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: test.mailbox
URL: <http://mail.python.org/pipermail/python-dev/attachments/20100629/53064c86/attachment-0007.ksh>