Pickle and exec in python

Question

¡Hi everybody!

What I have to do is this:

import pickle
exec "def f(): print 'Hi!'"
code = pickle.dumps(f)
print code

As you can see, it works fine.

But, in order to avoid defining global variables, (as for example f in previous code), I'd like to do something like this:

import pickle
d = {}
exec "def f(): print 'Hi!'" in d
d['f']()
code = pickle.dumps(d['f'])
print code

It prints 'Hi!', it means that python recognizes d['f'] as a function, but it can't pickle it.

Do you have any idea of how to do this? I think that it is because in the former case, python knew where to find the function (it is in '__main__') but in the latter case, it's inside a dictionary located in '__main__' I think that if you can tell python the real module direction of this function, it will work.

Thanks!

• Kevin: "In order to avoid defining global variables..." If you're hesitant to even define functions in the global scope, you may be taking that advice a bit too far. Anyway, d is global, so you're not really gaining much here.
• Sebastian Alvarez: What I mean is that I don't like that when I use the command exec(), it defines global variables (for example, I did'nt define f in the program. It was defined after using exec). That's why I have to keep all those variables in a dict. It's done by using exec code in dict
• Kevin: I suggest not using exec at all. Just write your def f(): regularly.
• Sebastian Alvarez: But in practice, the string will be replaced by any python code, and I don't know what it is, that's why I need to use exec.

Answer 1

pickle pickles functions by name. Rather than attempting to dump and restore the bytecode, it just says "okay, this function is __main__.f, so to unpickle it, look in the __main__ module and retrieve the thing called f". This is pretty much necessary; if you unpickle a function f in an interpreter environment where f doesn't exist or does something different, then to preserve the original behavior of the unpickled function, you'd need to somehow retain all the functions it references, all the modules it uses, all the global variables that don't exist any more, etc. You'd need to make sure that recursive references to f inside f use the unpickled function instead of the new interpreter's f, and it all becomes a horrible mess.

The point of that all is that pickled functions have to be global. If you look in the module specified by their __module__ and look for the thing named by their __name__, you have to find the function, or you can't unpickle it. Thus, you can't exec the function definition in a fake global dictionary, and exec-ing definitions of functions you want to pickle at all is a highly bug-prone maneuver.

Answer 2

I've found a very nice solution here: Dynamically importing Python module

And a code to solve the above problem is this:

import pickle
import sys
import imp
foo = imp.new_module("foo")
sys.modules["foo"] = foo
exec "def f(): print 'Hi!'" in foo.__dict__
code = pickle.dumps(foo.f)
print code

As you can see, the solution is to construct a new module and the function will be pickleable