lambda function not using the localmost scope

Question

I am writing a short code to process a data file and stumbled on something I don't understand in the way lambda functions work.

Here is the problem: I have a list of keywords, and a list of the indexes of the lines where those keywords appears in a data file, and want to apply certain operations to said lines (and/or the adjacent lines, hence the need for a list of indexes and not simply a list of lines).

To do so, I defined a dictionary associating every keyword with a lambda function that will apply the wanted operation to the wanted line. For example:

methnames = {'acell' : lambda i : float(dat[i][1]) } #with dat the data file

(except with multiple keywords, and more complex functions).

Now, to execute that, as I expected, it required to have a global variable named dat to be defined, so I just put a dat=[], as I would call those function from within a local scope where dat would be defined.

Except when I execute the whole code, I get an IndexError, and the traceback tells me that, even if the that lambda was indeed called from within a local scope where dat should normally be defined, it still uses the global dat.

Even if I could go around that, this seems a very strange behaviour for Python, so I am probably missing something.

Here is a simplified version of the code :

dat=[]
methnames = {'acell' : lambda i : float(dat[i][1]) }
def test(dat):
 return(methnames['acell'](0))
 a=test([['acell',0,1,1]])

which should normally give a=0, and here is the return:

Traceback (most recent call last):
 File "<ipython-input-21-cc8eb6df810c>", line 1, in <module>
runfile('/home/penwwern/Documents/mineralo/MinPhys/FrI/out/test.py', wdir='/home/penwwern/Documents/mineralo/MinPhys/FrI/out')
 File "/usr/lib/python3/dist-packages/spyderlib/widgets/externalshell/sitecustomize.py", line 699, in runfile
execfile(filename, namespace)
 File "/usr/lib/python3/dist-packages/spyderlib/widgets/externalshell/sitecustomize.py", line 88, in execfile
exec(compile(open(filename, 'rb').read(), filename, 'exec'), namespace)
 File "/home/penwwern/Documents/mineralo/MinPhys/FrI/out/test.py", line 18, in <module>
a=test([['acell',0,1,1]])
 File "/home/penwwern/Documents/mineralo/MinPhys/FrI/out/test.py", line 15, in test
 return(methnames['acell'](0))
 File "/home/penwwern/Documents/mineralo/MinPhys/FrI/out/test.py", line 9, in <lambda>
 methnames = {'acell' : lambda i : float(dat[i][1]) }
IndexError: list index out of range

• Moses Koledoye: Your list has only one item (another list) at index 0. Index 1 does not exist
• jonrsharpe: @MosesKoledoye that's not the list the lambda is accessing, though.
• penwy: @MosesKoledoye indeed, that was an error when I abridged the code, I corrected it and the problem is the same
• user2357112: "even if the that lambda was indeed called from within a local scope where dat should normally be defined, it still uses the global dat" - well, yeah. That's how functions work. They look up variables in the scope where they were defined, not the scope where they're called. People tried designing programming languages where functions look up variables in the scope where they're called, and it turns out that's a terrible idea.
• juanpa.arrivillaga: @Penwy it does, but of where the function is defined, not where it is called. At least, that is my understanding, but I'm trying to find this explicitly stated in the official documentation.

Answer 1

Python does not check the scope of a caller, but of where a function is defined. This because Python is lexically scoped.

In [1]: def f(): print(i)
In [2]: def f2():
 ...: i = 10
 ...: f()
 ...:
In [3]: f2()
---------------------------------------------------------------------------
NameError Traceback (most recent call last)
<ipython-input-3-fdec4c1c071f> in <module>()
----> 1 f2()
<ipython-input-2-5ab58f8d1867> in f2()
 1 def f2():
 2 i = 10
----> 3 f()
 4
<ipython-input-1-a5bb8b95e798> in f()
----> 1 def f(): print(i)
NameError: name 'i' is not defined

Now, we can bind i in the same scope where f is defined:

In [4]: i = 88
In [5]: f2()
88

However, it does check enclosing scopes with respect to where it is defined:

In [6]: def f3():
 ...: i = 1
 ...: def inner():
 ...: print(i)
 ...: return inner
 ...:
In [7]: inner = f3()
In [8]: inner()
1
In [9]: print(i)
88

Lexical scoping is pretty common. Here is some more information from wikipedia:

A fundamental distinction in scoping is what "part of a program" means. In languages with lexical scope (also called static scope), name resolution depends on the location in the source code and the lexical context, which is defined by where the named variable or function is defined. In contrast, in languages with dynamic scope the name resolution depends upon the program state when the name is encountered which is determined by the execution context or calling context. In practice, with lexical scope a variable's definition is resolved by searching its containing block or function, then if that fails searching the outer containing block, and so on, whereas with dynamic scope the calling function is searched, then the function which called that calling function, and so on, progressing up the call stack.[4] Of course, in both rules, we first look for a local definition of a variable.

https://en.wikipedia.org/wiki/Scope_(computer_science)#Lexical_scope_vs._dynamic_scope

• penwy: The problem was indeed my lack of understanding of the way scopes work. Thanks a lot.
• chepner: The technical terms of relevance here are lexical scope (which Python uses) and dynamic scope (which you were expecting).
• juanpa.arrivillaga: @chepner yes, was in the middle of adding that.