Why doesn't exec work in a function with a subfunction?

Question

It looks like you can't use exec in a function that has a subfunction...

Anyone know why this Python code doesn't work? I get an error at the exec in test2. Also, I know exec's aren't good style, but trust me, I'm using exec for an appropriate reason. I wouldn't use it otherwise.

#!/usr/bin/env python
#
def test1():
 exec('print "hi from test1"')
test1()
def test2():
 """Test with a subfunction."""
 exec('print "hi from test2"')
 def subfunction():
 return True
test2()

EDIT: I narrowed down the bug to having a function in a subfunction. It has nothing to do with the raise keyword.

• Sergey Orshanskiy: There is the same limitation with import * from ....
• abcd: looks to me like dict comprehensions are considered subfunctions.
• Ciro Santilli OurBigBook.com: @dbliss I reproduce on certain older Python versions: stackoverflow.com/a/52498591/895245

Answer 1

Correct. You can't use exec in a function that has a subfunction, unless you specify a context. From the docs:

If exec is used in a function and the function contains a nested block with free variables, the compiler will raise a SyntaxError unless the exec explicitly specifies the local namespace for the exec. (In other words, "exec obj" would be illegal, but "exec obj in ns" would be legal.)

There is good reason for this which I would probably understand if it wasn't Sunday night. Now, next question: Why are you using exec? It's very rarely needed. You say you have a good reason. I'm feeling sceptical about that. ;) If you have a good reason I'll tell you the workaround. :-P

Oh well, here it is anyway:

def test2():
 """Test with a subfunction."""
 exec 'print "hi from test2"' in globals(), locals()
 def subfunction():
 return True

• chiffa: Implementing a humane and object-oriented API for an esoteric NoSQL database that only has very low-level bindings to Python is an example. Though, an additional question for you: is there is a way to inject ''exec''-ed code into a specific class? If yes, how?
• Lennart Regebro: @AndreiKucharavy No, exec executes code, so there is no such thing as "exec"-ed code to inject. Execed code is executed code. You can compile code and inject that, maybe that's what you meant? But usually you don't need to compile it, you just inject it. There is no obvious reason why you have to exec anything just because you want an OO API to a NoSQL DB.
• chiffa: Sorry, my bad: not inject code, but insert it, so it is callable as being a part of specific class. The idea is to parse a user-created config file and from the definitions generate a family of objects that are callable as simple python objects, but under the hood implement a buffering interface with a database, a little bit like SQLAlchemy, but without the need to write all the mappings and object definition code explicitly.
• Lennart Regebro: @AndreiKucharavy I don't see how exec would help. Exec executes code. This does not seem to be what you want to do. You want to create classes dynamically. That's not what exec does. you should probably start a separate question. Or several.
• Søren Løvborg: For the benefits of other readers: The type function (three argument version) is the way to go for dynamically created classes.

Answer 2

Although in Python it looks kind of like the local variables are stored in a dictionary locals(), they usually aren't. Instead they are mostly stored on the stack and accessed by index. This makes local variable lookup faster than if it had to do a dictionary lookup every time. If you use the locals() function then what you get is a fresh dictionary created from all the local variables, and that's why assigning to locals() doesn't generally work.

There are a couple of exceptions to this scenario:

When you use an unqualified exec inside a function Python turns off the optimisation and uses a real dictionary for the local variables. That means you can create or update variables from inside the exec, but it also means all local variable access in that function will run more slowly.

The other exception is that when you nest functions the inner function can access local variables in the outer function scope. When it does this the variable is stored in a 'cell' object instead of being stored on the stack. The extra level of indirection makes all use of scoped variables slower whether you access them from the inner or outer function.

The catch that you've encountered is that these two exceptions to how local variables are normally stored are incompatible. You cannot have a variable stored in a dictionary and accessed through a cell reference at the same time. Python 2.x fixes this by disallowing the exec, even in cases like this where you aren't trying to use any scoped variables.

Answer 3

This is a rather interesting case:

>>> def func():
... exec('print "hi from func"')
... def subfunction():
... return True
... 
 File "<stdin>", line 2
SyntaxError: unqualified exec is not allowed in function 'func' because 
it contains a nested function with free variables

The reason why this doesn't work indeed is that subfunction contains a free variable, and since in Python 2, exec could theoretically modify the locals in the containing scope, it would be impossible to decide if the variable should be bound from the global or the parent function scope. One of the verses in the Zen of Python is "In the face of ambiguity, refuse the temptation to guess." and this is what Python 2 does.

Now the question is: what is this free (unbound) variable? Well, it is True!

Indeed it is reproduceable with None:

>>> def func():
... exec('print "hi from func"')
... def subfunction():
... return None
... 
 File "<stdin>", line 2
SyntaxError: unqualified exec is not allowed in function 'test2' because it contains a nested
function with free variables

Even though None cannot be assigned to, and it is considered as a constant in the bytecode, the buggy parser thinks it is an unbound variable here.

But if you replace it with 1 and it works without problems:

>>> def test2():
... exec('print "hi from func"')
... def subfunction():
... return 1
... 
>>>

To avoid this error, specify explicitly the globals and possibly locals that are to be used by exec, say:

>>> def test2():
... exec 'print "hi from test2"' in {}
... def subfunction():
... return None
...
>>>

---

In Python 3, exec is just a simple function and isn't handled specially by the parser or the bytecode compiler. In Python 3 exec cannot rebind function-local names, and thus this SyntaxError and ambiguity doesn't exist.

---

One peculiar case in Python 2 vs 3 compatibility is that while Python 2.7 documentation states that

The form exec(expr, globals) is equivalent to exec expr in globals, while the form exec(expr, globals, locals) is equivalent to exec expr in globals, locals. The tuple form of exec provides compatibility with Python 3, where exec is a function rather than a statement.

The tuple form has not always been 100 % compatible, as there was a bug in handling of exec in functions with nested functions (issue 21591); up to Python 2.7.8 the following code might have thrown an exception:

def func():
 exec('print "hi from test2"', {})
 def subfunction():
 return None

This was fixed in Python 2.7.9 and it no longer throws.

• wolfmanx: To avoid the unfortunate situtation of Python2/Python3 exec bugginess and compatibility, you can also use the idiom edict = {}; eval(compile(statement, '<string>', 'exec'), globals(), edict). This gives access to the side-effects of exec (see also this discussion of exec/eval/compile).

Answer 4

That works well in Python 3.1.3, after modifying the print statement to use print function.

In Python 2.6, it produces SyntaxError: unqualified exec is not allowed in function 'test2' it contains a nested function with free variables, I don't think it's a bug.

Answer 5

dict and list comprehensions may also be considered subfunctions on Python 2.7.5

For example, this fails on Python 2.7.5, but works on Python 2.7.12:

def func():
 exec('print("a")')
 (str(e) for e in range(10))

with:

 File "./a.py", line 4
 exec('print("a")')
SyntaxError: unqualified exec is not allowed in function 'func' it contains a nested function with free variables

Likely it got internally compiled to a function in the bytecode.

TODO find the fixing commit. It was beyond my git log --grep foo.

Analogous for dict comprehensions:

def func():
 exec('print("a")', {e:str(e) for e in range(10)})

which is specially bad since it is a common parameter for the global argument.

Also raised at: https://github.com/sphinx-doc/sphinx/issues/5417#issuecomment-421731085

Answer 6

The error seems to be fairly obvious to me:

SyntaxError: unqualified exec is not allowed in function 'test2' it contains a nested function with free variables

See pep 227 for more info: http://www.python.org/dev/peps/pep-0227/