Testing for performance regressions

Tue Apr 5 01:25:43 EDT 2011

On 2011年4月04日 20:59:52 -0700, geremy condra wrote:
> On Mon, Apr 4, 2011 at 7:45 PM, Steven D'Aprano
> <steve+comp.lang.python at pearwood.info> wrote:

>> * The disclaimers about timing code snippets that can be found in the
>> timeit module apply. If possible, use timeit rather than roll-you-own
>> timers.
>> Huh. In looking into timing attacks actually one of the biggest lessons
> I learned was *not* to use timeit- that the overhead and variance
> involved in using it will wind up consuming small changes in behavior in
> ways that are fairly opaque until you really take it apart.

Do you have more details?
I follow the advice in the timeit module, and only ever look at the 
minimum value, and never try to calculate a mean or variance. Given the 
number of outside influences ("What do you mean starting up a browser 
with 200 tabs at the same time will affect the timing?"), I wouldn't 
trust a mean or variance to be meaningful.
-- 
Steven