homepage

-locals() returns object that does't walk like a duck.
+locals() returns object that does't work like a dict.
Much of the confusion with locals() comes from the fact that returned object is labelled a dict, but this heisendict changes its behavior when you're trying to introspect it. So, to give users a hint that this object is different, I propose to rename it to a different type and supply an appropriate help() for it.

The documentation seems pretty clear on the nature of the dict returned by locals(). [1] What documentation are you referring to?
[1] http://docs.python.org/3.4/library/functions.html#locals 

Sorry, didn't mean to change the attributed version. However the 2.7 documentation says the same thing.

1. The documentation doesn't say that the content of this dict may change sporadically even if you don't call locals()
2. I am referring to inline documentation:
 >>> h = locals()
 >>> type(h)
 <class 'dict'>
 >>> help(h)

So the problem you are having is that inside functions the dict returned by locals() does not get updated when the "current local symbol table" changes? Keep in mind that for modules and classes (the other two execution blocks), the dict returned by locals() does get updated.
This difference reflects that fact that, in CPython at least, the local execution namespace of a function call's stack frame is not stored in a dictionary. When you call locals() in a function it is only making a copy of the frame's fast locals. The fast locals never gets directly exposed except through normal name lookup/binding/deletion. While technically possible, changing this simply isn't worth it.
The documentation already makes it clear that the dict returned by locals() represents the "current local symbol table" and that you should not expect changes to that dict to be reflected in the actual locals. Are you recommending that it be more clear that the dict may be no more than a snapshot of the locals at the time locals() is called?

Under the trace function, the dict is always updated, and that changed a workflow in the program I was debugging leading to heisenbug. But that's a different story. I'd like to concentrate on the reasons to rename locals() result type from 'dict' to 'livedict' (or other internal type with well-defined behavior).
dict() returned by locals() is sporadically updated. The problem is not with when it is updated, or how it is updated. The problem is that the fact that it is updated is not indicated in any way. And the problem that comes from this problem is that people don't realize that this dict may change its contents. As a result people become confused with Python.
The documentation is also ambiguous. Those people who don't know about self-updating behavior (I was) read "represents" like a "copy of symbol table", not like a "internally updated view of symbol table".
From the OOP point of view, the object doesn't behave like a normal dict, and it will be less confusing if it is renamed.
offtopic: "local execution namespace of a function call's stack frame" sounds cryptic - I am afraid that without visualization of local/global/namespace/scope/frame I am unable to understand that right now.
offtopic2: I also don't get "you should not expect changes to that dict to be reflected in the actual locals". I reads as if in some cases modifying the dict will change the locals. I thought that it is impossible as it is just a copy.

Thanks for the details, Anatoly. I am surprised by the behavior you've described. It may be that functions don't use fast locals when tracing is turned on. I'll have to check. If that's the case, I think the documentation for locals() should be improved. Either way, I agree that inconsistent (and undocumented) behavior is a headache. However, changing the type of the object returned by locals() is going to take a lot more justification. A doc update should be sufficient and a much easier sell.
Here are some further questions:
1. the dynamic update you described happens in function bodies (i.e. stack frames)?
2. does that behavior happen when you are not using a tracing function?
Also, if you have a minute, throw up a patch that reproduces the behavior you're talking about. That will help get this resolved faster.
Keep in mind that locals() always returns a normal dict. Just like any other dict, other code that has a reference to that "locals" dict can interact with it, which is what you have described. In this case, the interpreter is doing so. The only case where I find that surprising is where a stack frame is using fast locals, which is what function calls normally do.

Raymond, could you please get the title back? You unintentionally hijacked the issue. The "Document the circumstances where the locals() dict gets updated" is not a describing title for this issue - it is one of the possible action items. I'd prefer to see a separate issue to document the stuff that "blocks" this one. It is quite evident that this action is required, but IMHO there is not enough evidence to conclude that it will be sufficient to close this ticket.
Eric, I need more time to answer your questions.

The previous title, "rename type returned by locals() to livedict" did not describe the reality, since locals() returns a regular dict.
[Would you call x.__dict__ a livedict?]
So either this issue should be closed as invalid, because it's based on incorrect understandings of Python internals; or we could improve documentation about locals() in order to remove surprising behavior and (real) confusion in programmers' minds.

Eric, here is the code that confirms changed behavior under trace function for both Python 2 and 3 - http://bugs.python.org/file15081/localstest.py
I agree that the documentation fix is necessary, and if you say that it is an easier sell - I tend to believe you. But with this fix the issue is still not fully covered, so it is interesting to know what builds up to the cost of renaming if we come to some common point about the type of this 'dict' from the user point of view.

Amaury, from user's point of view (I am not a core developer - I just need to troubleshoot complicated Python code) the object (the internal structure) returned by locals() has different behavior than a normal dict.
Normal dict in Python is updated by user code (which "I as a user" can see and can inspect for further troubleshooting) and for locals's dict this is not correct. 
If dict and locals's object are of the same type, I'd expect them to be interchangeable.
By proposing a documentation fix as a final remedy, please think about the entry point through which users come to this "trap".

> Normal dict in Python is updated by user code (which "I as a user" can
> see and can inspect for further troubleshooting) and for locals's dict
> this is not correct.
Do you have an example?

Example:
 l = locals()
 z = dict(a=5, b=3)
 lc = dict(l)
 zc = dict(z)
 print(lc == l)
 print(zc == z)
Gives:
 False
 True

On 3/27/2013 9:48 AM, anatoly techtonik wrote:
> anatoly techtonik added the comment:
>
> Example:
>
> l = locals()
> z = dict(a=5, b=3)
>
> lc = dict(l)
> zc = dict(z)
>
> print(lc == l)
> print(zc == z)
>
> Gives:
>
> False
> True
Expected behavior.
lc is not another reference to locals(). It refers to a new object.
zc = ... updates locals() but not lc

This is expected: "zc = xxx" updates locals!
but not the copy.

> "I as a user" can
> see and can inspect for further troubleshooting
This doesn't work for this example, which was your question.
I still feel like this needs further clarification. For a user this behavior is not expected. User is someone who doesn't know the internal details of Python interpreter (and may not even remember that dict can be modified by reference). If you disagree with me - try to do corridor testing and see how many people will compile it in their head and give you the right answer to this example.

Thanks for the script Anatoly. That's pretty much what I was imagining from your description. This definitely reinforces my belief that profiling.

Okay, I found it. sys.settrace() ultimately results in the setting of tstate->use_tracing to true and sets tstate->c_tracefunc and tstate_c_traceobj (see sys_settrace() in Python/sysmodule.c and PyEval_SetTrace() in Python/ceval.c). tstate->c_tracefunc() gets set to trace_trampoline() in Python/sysmodule.c and tstate->c_traceobj gets set to your tracing function.
When an execution frame begins, the interpreter sets up tracing. From Python/ceval.c:1124:
/* tstate->c_tracefunc, if defined, is a
 function that will be called on *every* entry
 to a code block. Its return value, if not
 None, is a function that will be called at
 the start of each executed line of code.
 (Actually, the function must return itself
 in order to continue tracing.) The trace
 functions are called with three arguments:
 a pointer to the current frame, a string
 indicating why the function is called, and
 an argument which depends on the situation.
 The global trace function is also called
 whenever an exception is detected. */
So trace_trampoline() gets at the start of each block and once for each line of Python code. Each time it calls call_trampoline() (also in Python/sysmodule.c). You'll find that in call_trampoline(), there is a call to PyFrame_FastToLocals() just before it calls your tracing function (at that point called "callback").
When called, PyFrame_FastToLocals() updates the contents of the frame's "slow" locals (f_locals) with the values in the fast locals. And...wait for it...f_locals is the dict that gets returned by locals().
Thus, when tracing is on, the dict returned by locals() gets updated once per block and once per line of Python code. That is exactly what you are seeing.
When tracing is not on, PyFrame_FastToLocals() would only be called when you call locals() (inside a function). It's interesting to note that in that case locals() will return the exact same dictionary:
>>> def f():
... return locals() is locals()
...
>>> f()
True
Conclusion
----------
The bottom line is that the docs for locals() could stand to have a little more detail for this case (including a link to the docs for tracing at sys.settrace() or wherever). However, the behavior here is--at present--a CPython implementation detail. Any note would likely say as much; something like this:
.. function:: locals()
 ...
 Each call to locals() will return the same dictionary, updated to
 the contents of the current local symbol table.
 .. impl-detail::
 Under tracing and profiling in CPython, the dict returned by
 ``locals()`` will be updated at the beginning of each code block
 and at each line of code. See :func:`sys.settrace`.
The note is really only meaningful for functions since class bodies and modules don't use fast locals and f_locals is set to f_globals (i.e. locals() == globals()). However, that point is superfluous to the above note since it remains true either way.

I should also point out that the following from that note may actually also be a CPython implementation detail:
 Each call to locals() will return the same dictionary, updated to
 the contents of the current local symbol table.
We'd want to be sure that such should be expected from all implementations (since it basically adds to the language specification). Otherwise it should also go in an impl-detail note.

I've indicated that this issue supercedes issue #7083.

It could be the implementation detail if locals() calls were not useful to majority of Python software, and if behavior of its returned dict did not affect the execution flow of user programs (was deterministic for users who should not care about the implementation details).
What I proposed initially is to wrap the locals(), the dict it returns and various calls that update this dict into a pattern called "livedict". This pattern has a defined behavior, and in OOP it would be encapsulated in a single object with one private method locals.__update (called by FastToLocals and other internals), and accessor method locals.todict() (name may vary) to receive a static copy of the live cache. If somebody needs access to real-time updated symbol table, it may request to work with realtime API of this locals object instead. This API is turn could have an implementation detail that transparently provided faster access to the local symbol table than copying dict into cache every time.
Wrapping this into a pattern also makes it easy to communicate details in case other Python implementation breaks this implementation detail of CPython. It may be easier for understanding to start with broken aspect of the pattern without digressing into implementation details, and read about implementation details later, than to go other way around.

locals() being useful to the majority of Python software is news to me...

Changing the type of the locals() return isn't the right way to solve this problem. Better docs is the way to do it. If someone is calling locals(), they should read the docs. Examining the type of values is discouraged in Python, we shouldn't expect people to discover nonintuitive behavior because the type has been artificially changed to "livedict".

> Changing the type of the locals() return isn't the right way to solve
> this problem. Better docs is the way to do it. If someone is calling
> locals(), they should read the docs. Examining the type of values is
> discouraged in Python, we shouldn't expect people to discover
> nonintuitive behavior because the type has been artificially changed
> to "livedict".
+1
Issue #7083 was created to promote a semantic change (ultimately) and it was rejected in deference to a doc change. That's part of why this ticket was changed to focus on documentation.
My only concern is that locals() has some pretty heavy implementation specifics, so any doc change needs to respect that (or be part of an effort to tighten the spec for locals()).

Ok. Maybe I proposed the solution too early. I am fine with doc fix, but given the previous experience I have doubts it will happen. Maybe opening a wiki page with a current draft can help:
 http://wiki.python.org/moin/Locals
 or better
 http://piratepad.net/locals
Describing implementation details is necessary, but they should include a recommendation how to work with locals safely and avoid hitting yourself in the foot if you don't want to read about internals. For that I propose "if you are not interested in implementation details, always copy locals() to dict to avoid potential problems with auto-updating behavior. This is also guaranteed to work with other Python implementations." Is that ok?

After the last exchange on #7083, Guido emailed me that I could/should revised the doc a bit, but he said he did *not* want the details explained because he regards them as CPython implementation details and not part of the language definition. I have not done so yet because of other issues, especially the Idle PEP and related discussion. What is needed is something like
"The relationship between the local namespace and the locals dict is undefined between calls to locals(). Changes to either may or may not be propagated to the other."
Perhaps add "The dict returned by locals() is intended to be used immediately, either for introspection or for passing on to functions such as eval() and exec()." and even "It is not intended to be saved and used later."
What I have not figured out yet is how to integrate the above with the current text.

> them as CPython implementation details and not part of the language definition
What is the formal language definition of locals() then? What behavior and functionality of locals() should be supported by all Python implementations? For a potential language developer, what should locals() calls return to be compatible with this Python definition?

I would say it something like this:
"The dictionary returned by locals() is an accurate snapshot of the local namespace at the time it is called. After the call returns, changes to one may or may not be reflected in the other. The dictionary may change unpredictably after the call, and how it does is implementation-specific. Holding a reference to the dictionary may produce surprising effects."

I also think 'free variable' should be replace by 'nonlocal name'.

Terry: agreed.

The problem with using 'nonlocal' is that it would naturally be read as 'those names referenced in a nonlocal statement', which I don't think is the case. Maybe 'non-local', to differentiate it from the keyword?
I'm attaching a patch incorporating my understanding of the suggestions made so far.

/non-local/Non-local/ If we add the hyphen, perhaps we should add 'non-global' also.
Thanks for the concrete draft. It is definitely an improvement.
The last sentence is somewhat redundant with the second. The problem with the last sentence is that it only said that changes to the dict might not propagate back the the namespace, while is is also true that changes to the namespace might or might not propagate to the dict. The new second sentence says both. In addition, I somewhat disagree with 'should not modify'. A person could have a perfectly good reason to do so, but they should not expect propagation of the changes. So I am tempted to drop the last sentence. On the other hand, 'undefined' should be somewhere.

Hmm. Perhaps the last sentence could be "... because changes to the local dict propagating to the local namespace cannot be relied upon to either happen or not happen". That would make it less redundant, since it would essentially be referencing the previous statement in the specific case of the consequences of modification.
The original included the caution against modifying it, and I think it is valid because of the inconsistent behavior. Perhaps it could be weakened to "it is not a good idea to modify"?

> ... cannot be relied upon to either happen or not happen...
IMHO this phrase is from Advanced English course.
> The original included the caution against modifying it, and I think it is
valid because of the inconsistent behavior.
> Perhaps it could be weakened to "it is not a good idea to modify"?
Ambiguity already adds fear and uncertainty. Just explaining the
consequences would be good style for the technical doc.

Combined the second and last lines, discarded duplication.

Your formulation is more concise, thank you.
I suggest dropping the word 'additionally'. Also, "how it does" would be better phrased as "how it changes", I think. (It really should be "whether and how it changes", but in deference to Anatoly's 'advanced English' comment I'm willing to let that imprecision slide).

What about instead of
'''
Whether changes to one are reflected in the other after the call returns is undefined; additionally, the dictionary may change unpredictably after the call, and how it does is implementation-specific.
'''
substitue this wording:
'''
Whether changes to one are reflected in the other after the call returns, and when such updates occur, is undefined and implementation-specific.
'''
The old wording seems under-specified. It would allow a function call, garbage collection, etc, to clobber the dictionary, say overwriting with another function’s locals(), before you get a chance to work with the dictionary.

Yeah, the question of thread-safety in regards to what we are talking about here also occurred to me. That is, the wording makes one wonder if locals is thread safe or not. I don't see your suggested wording as making it clearer, though.
The problem is that it *is* underspecified.
So I think the correct description of the current under-specification is that locals() returns a copy of the current locals namespace. If you modify the thing returned by locals it may or may not update the local namespace. If you modify the local namespace (by doing x = y or its equivalents) the change may or may not be reflected in the object that was returned by locals().
Now, how do we boil that down? Or do we just say it more or less that way?

Here is another attempt with different words:
'''
.. note::
 The dictionary returned by :func:`locals` is an accurate snapshot of the local namespace at the time it is called. If the namespace changes after the call, the dictionary may become out of date, but it may also automatically update at any time. The contents of the dictionary should not be modified by the user; it is undefined whether such changes affect the namespace or not.
'''

I mostly like Martin's suggested wording, but would also note that I filed issue 17960 to tighten up the requirements for when we expect assigning to locals() to work.
To save folks reading the whole referenced email, I think it would be worth defining that modifying the namespace returned by locals() will affect the runtime namespace at module and class scope and when using exec, but will have no effect at function scope (as locals() returns a copy of the namespace in that case).

I quickly scanned through the email thread from Issue 17960. I guess it makes sense to specify that locals() can be used to directly get a class’s namespace. Probably doesn’t hurt to say locals() is equivalent to globals() at module level, although this seems like a fairly redundant feature.
Here is locals_doc.03.patch, which uses my wording for function namespaces, and also adds more for class and global namespaces, as suggested by Nick.

Some minor tweaks to my earlier patch:
* list comprehension → comprehension
* time it is called → time of the call