unit-profiling, similar to unit-testing

Thu Nov 17 21:00:00 EST 2011

In article <mailman.2810.1321562763.27778.python-list at python.org>,
 Tycho Andersen <tycho at tycho.ws> wrote:
> While I agree there's a lot of things you can't control for, you can
> get a more accurate picture by using CPU time instead of wall time
> (e.g. the clock() system call). If what you care about is mostly CPU
> time [...]

That's a big if. In some cases, CPU time is important, but more often, 
wall-clock time is more critical. Let's say I've got two versions of a 
program. Here's some results for my test run:
Version CPU Time Wall-Clock Time
 1 2 hours 2.5 hours
 2 1.5 hours 5.0 hours
Between versions, I reduced the CPU time to complete the given task, but 
increased the wall clock time. Perhaps I doubled the size of some hash 
table. Now I get a lot fewer hash collisions (so I spend less CPU time 
re-hashing), but my memory usage went up so I'm paging a lot and my 
locality of reference went down so my main memory cache hit rate is 
worse.
Which is better? I think most people would say version 1 is better.
CPU time is only important in a situation where the system is CPU bound. 
In many real-life cases, that's not at all true. Things can be memory 
bound. Or I/O bound (which, when you consider paging, is often the same 
thing as memory bound). Or lock-contention bound.
Before you starting measuring things, it's usually a good idea to know 
what you want to measure, and why :-)