Return value while using cProfile

Question

I'm trying to profile an instance method, so I've done something like:

import cProfile
class Test():
 def __init__(self):
 pass
 def method(self):
 cProfile.runctx("self.method_actual()", globals(), locals())
 def method_actual(self):
 print "Run"
if __name__ == "__main__":
 Test().method()

But now problems arise when I want "method" to return a value that is computed by "method_actual". I don't really want to call "method_actual" twice.

Is there another way, something that can be thread safe? (In my application, the cProfile data are saved to datafiles named by one of the args, so they don't get clobbered and I can combine them later.)

Answer 1

I discovered that you can do this:

prof = cProfile.Profile()
retval = prof.runcall(self.method_actual, *args, **kwargs)
prof.dump_stats(datafn)

The downside is that it's undocumented.

• Jonathan Hartley: Brilliant! This looks perfect - but what is 'datafn'?
• detly: @JonathanHartley - The filename for the data file IIRC.
• Jonathan Hartley: Ah, thanks. I thought 'fn' meant function, not filename.
• detly: @Ngoral it's just a file name. You can put whatever name you like there.
• detly: @Ngoral no worries, I should clarify in the answer

Answer 2

An option for any arbitrary code:

import cProfile, pstats, sys
pr = cProfile.Profile()
pr.enable()
my_return_val = my_func(my_arg)
pr.disable()
ps = pstats.Stats(pr, stream=sys.stdout)
ps.print_stats()

Taken from https://docs.python.org/2/library/profile.html#profile.Profile

• detly: You could even make a little context manager for that using contextlibs contextmanager decorator.
• Mr_and_Mrs_D: I get Random listing order was used - how can I specify listing order ?
• marsh: ps.sort_stats('cumulative')
• Ishan Srivastava: how do I generate a valid profile file with this method so that I can use it with SnakeViz
• Kevin Newman: I would only add something like: ps.dump_stats('my_dumpfile.dmp') so one can use snakeviz or similar to view the results, otherwise this worked great for me! I was having a heckuva time getting my return values before this.

Answer 3

I was struggling with the same problem and used a wrapper function to get over direct return values. Instead of

cP.runctx("a=foo()", globals(), locales())

I create a wrapper function

def wrapper(b):
 b.append(foo())

and profile the call to the wrapper function

b = []
cP.runctx("wrapper(b)", globals(), locals())
a = b[0]

extracting the result of foo's computation from the out param (b) afterwards.

Answer 4

I created a decorator:

import cProfile
import functools
import pstats
def profile(func):
 @functools.wraps(func)
 def inner(*args, **kwargs):
 profiler = cProfile.Profile()
 profiler.enable()
 try:
 retval = func(*args, **kwargs)
 finally:
 profiler.disable()
 with open('profile.out', 'w') as profile_file:
 stats = pstats.Stats(profiler, stream=profile_file)
 stats.print_stats()
 return retval
 return inner

Decorate your function or method with it:

@profile
def somefunc(...):
 ...

Now that function will be profiled.

Alternatively, if you'd like the raw, unprocessed profile data (e.g. because you want to run the excellent graphical viewer RunSnakeRun on it), then:

import cProfile
import functools
import pstats
def profile(func):
 @functools.wraps(func)
 def inner(*args, **kwargs):
 profiler = cProfile.Profile()
 profiler.enable()
 try:
 retval = func(*args, **kwargs)
 finally:
 profiler.disable()
 profiler.dump_stats('profile.out')
 return retval
 return inner

This is a minor improvement on several of the other answers on this page.

Answer 5

I think @detly the .runcall() is basically the best answer, but for completeness, I just wanted to take @ThomasH 's answer to be function independent:

def wrapper(b, f, *myargs, **mykwargs):
 try:
 b.append(f(*myargs, **mykwargs))
 except TypeError:
 print 'bad args passed to func.'
# Example run
def func(a, n):
 return n*a + 1
b = []
cProfile.runctx("wrapper(b, func, 3, n=1)", globals(), locals())
a = b[0]
print 'a, ', a